Index: head/en_US.ISO8859-1/articles/committers-guide/article.xml =================================================================== --- head/en_US.ISO8859-1/articles/committers-guide/article.xml (revision 47723) +++ head/en_US.ISO8859-1/articles/committers-guide/article.xml (revision 47724) @@ -1,5011 +1,5010 @@ ]>
Committer's Guide The &os; Documentation Project 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 The &os; Documentation Project &tm-attrib.freebsd; &tm-attrib.coverity; &tm-attrib.ibm; &tm-attrib.intel; &tm-attrib.sparc; &tm-attrib.general; $FreeBSD$ $FreeBSD$ This document provides information for the &os; committer community. All new committers should read this document before they start, and existing committers are strongly encouraged to review it from time to time. Almost all &os; developers have commit rights to one or more repositories. However, a few developers do not, and some of the information here applies to them as well. (For instance, some people only have rights to work with the Problem Report database). Please see for more information. This document may also be of interest to members of the &os; community who want to learn more about how the project works. Administrative Details Login Methods &man.ssh.1;, protocol 2 only Main Shell Host freefall.FreeBSD.org src/ Subversion Root svn+ssh://repo.FreeBSD.org/base (see also ). doc/ Subversion Root svn+ssh://repo.FreeBSD.org/doc (see also ). ports/ Subversion Root svn+ssh://repo.FreeBSD.org/ports (see also ). Internal Mailing Lists developers (technically called all-developers), doc-developers, doc-committers, ports-developers, ports-committers, src-developers, src-committers. (Each project repository has its own -developers and -committers mailing lists. Archives for these lists may be found in files /home/mail/repository-name-developers-archive and /home/mail/repository-name-committers-archive on the FreeBSD.org cluster.) Core Team monthly reports /home/core/public/monthly-reports on the FreeBSD.org cluster. Ports Management Team monthly reports /home/portmgr/public/monthly-reports on the FreeBSD.org cluster. Noteworthy src/ SVN Branches stable/8 (8.X-STABLE), stable/9 (9.X-STABLE), stable/10 (10.X-STABLE), head (-CURRENT) &man.ssh.1; is required to connect to the project hosts. For more information, see . Useful links: &os; Project Internal Pages &os; Project Hosts &os; Project Administrative Groups Open<acronym>PGP</acronym> Keys for &os; Cryptographic keys conforming to the OpenPGP (Pretty Good Privacy) standard are used by the &os; project to authenticate committers. Messages carrying important information like public SSH keys can be signed with the OpenPGP key to prove that they are really from the committer. See PGP & GPG: Email for the Practical Paranoid by Michael Lucas and for more information. Creating a Key Existing keys can be used, but should be checked with doc/head/share/pgpkeys/checkkey.sh first. For those who do not yet have an OpenPGP key, or need a new key to meet &os; security requirements, here we show how to generate one. Install security/gnupg. Enter these lines in ~/.gnupg/gpg.conf to set minimum acceptable defaults: fixed-list-mode keyid-format 0xlong personal-digest-preferences SHA512 SHA384 SHA256 SHA224 default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 BZIP2 ZLIB ZIP Uncompressed use-agent verify-options show-uid-validity list-options show-uid-validity sig-notation issuer-fpr@notations.openpgp.fifthhorseman.net=%g cert-digest-algo SHA512 Generate a key: &prompt.user; gpg --full-gen-key gpg (GnuPG) 2.1.8; Copyright (C) 2015 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Warning: using insecure memory! Please select what kind of key you want: (1) RSA and RSA (default) (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) Your selection? 1 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) 2048 Requested keysize is 2048 bits Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) 3y Key expires at Wed Nov 4 17:20:20 2015 MST Is this correct? (y/N) y GnuPG needs to construct a user ID to identify your key. Real name: Chucky Daemon Email address: notreal@example.com Comment: You selected this USER-ID: "Chucky Daemon <notreal@example.com>" Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o You need a Passphrase to protect your secret key. 2048-bit keys with a three-year expiration provide adequate protection at present (2013-12). describes the situation in more detail. A three year key lifespan is short enough to obsolete keys weakened by advancing computer power, but long enough to reduce key management problems. Use your real name here, preferably matching that shown on government-issued ID to make it easier for others to verify your identity. Text that may help others identify you can be entered in the Comment section. After the email address is entered, a passphrase is requested. Methods of creating a secure passphrase are contentious. Rather than suggest a single way, here are some links to sites that describe various methods: , , , . Protect your private key and passphrase. If either the private key or passphrase may have been compromised or disclosed, immediately notify accounts@FreeBSD.org and revoke the key. Committing the new key is shown in . Kerberos and LDAP web Password for &os; Cluster The &os; cluster requires a Kerberos password to access certain services. The Kerberos password also serves as the LDAP web password, since LDAP is proxying to Kerberos in the cluster. Some of the services which require this include: Bugzilla Jenkins To reset a Kerberos password in the &os; cluster using a random password generator: &prompt.user; ssh kpasswd.freebsd.org This must be done from a machine outside of the &os;.org cluster. A Kerberos password can also be set manually by logging into freefall.FreeBSD.org and running: &prompt.user; kpasswd Commit Bit Types The &os; repository has a number of components which, when combined, support the basic operating system source, documentation, third party application ports infrastructure, and various maintained utilities. When &os; commit bits are allocated, the areas of the tree where the bit may be used are specified. Generally, the areas associated with a bit reflect who authorized the allocation of the commit bit. Additional areas of authority may be added at a later date: when this occurs, the committer should follow normal commit bit allocation procedures for that area of the tree, seeking approval from the appropriate entity and possibly getting a mentor for that area for some period of time. Committer Type Responsible Tree Components src core@ src/, doc/ subject to appropriate review doc doceng@ doc/, ports/, src/ documentation ports portmgr@ ports/ Commit bits allocated prior to the development of the notion of areas of authority may be appropriate for use in many parts of the tree. However, common sense dictates that a committer who has not previously worked in an area of the tree seek review prior to committing, seek approval from the appropriate responsible party, and/or work with a mentor. Since the rules regarding code maintenance differ by area of the tree, this is as much for the benefit of the committer working in an area of less familiarity as it is for others working on the tree. Committers are encouraged to seek review for their work as part of the normal development process, regardless of the area of the tree where the work is occurring. Policy for Committer Activity in Other Trees All committers may modify base/head/share/misc/committers-*.dot, base/head/usr.bin/calendar/calendars/calendar.freebsd, and ports/head/astro/xearth/files. doc committers may commit documentation changes to src files, such as man pages, READMEs, fortune databases, calendar files, and comment fixes without approval from a src committer, subject to the normal care and tending of commits. Any committer may make changes to any other tree with an "Approved by" from a non-mentored committer with the appropriate bit. Committers can aquire an additional bit by the usual process of finding a mentor who will propose them to core, doceng, or portmgr, as appropriate. When approved, they will be added to 'access' and the normal mentoring period will ensue, which will involve a continuing of Approved by for some period. "Approved by" is only acceptable from non-mentored src committers -- mentored committers can provide a "Reviewed by" but not an "Approved by". Subversion Primer It is assumed that you are already familiar with the basic operation of Subversion. If not, start by reading the Subversion Book. Introduction The &os; source repository switched from CVS to Subversion on May 31st, 2008. The first real SVN commit is r179447. The &os; doc/www repository switched from CVS to Subversion on May 19th, 2012. The first real SVN commit is r38821. The &os; ports repository switched from CVS to Subversion on July 14th, 2012. The first real SVN commit is r300894. Subversion can be installed from the &os; Ports Collection by issuing these commands: &prompt.root; pkg install subversion Getting Started There are a few ways to obtain a working copy of the tree from Subversion. This section will explain them. Direct Checkout The first is to check out directly from the main repository. For the src tree, use: &prompt.user; svn checkout svn+ssh://repo.freebsd.org/base/head /usr/src For the doc tree, use: &prompt.user; svn checkout svn+ssh://repo.freebsd.org/doc/head /usr/doc For the ports tree, use: &prompt.user; svn checkout svn+ssh://repo.freebsd.org/ports/head /usr/ports Though the remaining examples in this document are written with the workflow of working with the src tree in mind, the underlying concepts are the same for working with the doc and the ports tree. Ports related Subversion operations are listed in . The above command will check out a CURRENT source tree as /usr/src/, which can be any target directory on the local filesystem. Omitting the final argument of that command causes the working copy, in this case, to be named head, but that can be renamed safely. svn+ssh means the SVN protocol tunnelled over SSH. The name of the server is repo.freebsd.org, base is the path to the repository, and head is the subdirectory within the repository. If your &os; login name is different from your login name on your local machine, you must either include it in the URL (for example svn+ssh://jarjar@repo.freebsd.org/base/head), or add an entry to your ~/.ssh/config in the form: Host repo.freebsd.org User jarjar This is the simplest method, but it is hard to tell just yet how much load it will place on the repository. The svn diff does not require access to the server as SVN stores a reference copy of every file in the working copy. This, however, means that Subversion working copies are very large in size. Checkout from a Mirror Check out a working copy from a mirror by substituting the mirror's URL for svn+ssh://repo.freebsd.org/base. This can be an official mirror or a mirror maintained by using svnsync. There is a serious disadvantage to this method: every time something is to be committed, a svn relocate to the master repository has to be done, remembering to svn relocate back to the mirror after the commit. Also, since svn relocate only works between repositories that have the same UUID, some hacking of the local repository's UUID has to occur before it is possible to start using it. The hassle of a local svnsync mirror probably is not worth it unless the network connectivity situation or other factors demand it. If it is needed, see the end of this chapter for information on how to set one up. <literal>RELENG_*</literal> Branches and General Layout In svn+ssh://repo.freebsd.org/base, base refers to the source tree. Similarly, ports refers to the ports tree, and so on. These are separate repositories with their own change number sequences, access controls and commit mail. For the base repository, HEAD refers to the -CURRENT tree. For example, head/bin/ls is what would go into /usr/src/bin/ls in a release. Some key locations are: /head/ which corresponds to HEAD, also known as -CURRENT. /stable/n which corresponds to RELENG_n. /releng/n.n which corresponds to RELENG_n_n. /release/n.n.n which corresponds to RELENG_n_n_n_RELEASE. /vendor* is the vendor branch import work area. This directory itself does not contain branches, however its subdirectories do. This contrasts with the stable, releng and release directories. /projects and /user feature a branch work area, like in Perforce. As above, the /user directory does not contain branches itself. &os; Documentation Project Branches and Layout In svn+ssh://repo.freebsd.org/doc, doc refers to the repository root of the source tree. In general, most &os; Documentation Project work will be done within the head/ branch of the documentation source tree. &os; documentation is written and/or translated to various languages, each in a separate directory in the head/ branch. Each translation set contains several subdirectories for the various parts of the &os; Documentation Project. A few noteworthy directories are: /articles/ contains the source code for articles written by various &os; contributors. /books/ contains the source code for the different books, such as the &os; Handbook. /htdocs/ contains the source code for the &os; website. &os; Ports Tree Branches and Layout In svn+ssh://repo.freebsd.org/ports, ports refers to the repository root of the ports tree. In general, most &os; port work will be done within the head/ branch of the ports tree which is the actual ports tree used to install software. Some other key locations are: /branches/RELENG_n_n_n which corresponds to RELENG_n_n_n is used to merge back security updates in preparation for a release. /tags/RELEASE_n_n_n which corresponds to RELEASE_n_n_n represents a release tag of the ports tree. /tags/RELEASE_n_EOL represents the end of life tag of a specific &os; branch. Daily Use This section will explain how to perform common day-to-day operations with Subversion. Help SVN has built in help documentation. It can be accessed by typing the following command: &prompt.user; svn help Additional information can be found in the Subversion Book. Checkout As seen earlier, to check out the &os; head branch: &prompt.user; svn checkout svn+ssh://repo.freebsd.org/base/head /usr/src At some point, more than just HEAD will probably be useful, for instance when merging changes to stable/7. Therefore, it may be useful to have a partial checkout of the complete tree (a full checkout would be very painful). To do this, first check out the root of the repository: &prompt.user; svn checkout --depth=immediates svn+ssh://repo.freebsd.org/base This will give base with all the files it contains (at the time of writing, just ROADMAP.txt) and empty subdirectories for head, stable, vendor and so on. Expanding the working copy is possible. Just change the depth of the various subdirectories: &prompt.user; svn up --set-depth=infinity base/head &prompt.user; svn up --set-depth=immediates base/release base/releng base/stable The above command will pull down a full copy of head, plus empty copies of every release tag, every releng branch, and every stable branch. If at a later date merging to 7-STABLE is required, expand the working copy: &prompt.user; svn up --set-depth=infinity base/stable/7 Subtrees do not have to be expanded completely. For instance, expanding only stable/7/sys and then later expand the rest of stable/7: &prompt.user; svn up --set-depth=infinity base/stable/7/sys &prompt.user; svn up --set-depth=infinity base/stable/7 Updating the tree with svn update will only update what was previously asked for (in this case, head and stable/7; it will not pull down the whole tree. Decreasing the depth of a working copy is not possible. Anonymous Checkout It is possible to anonymously check out the &os; repository with Subversion. This will give access to a read-only tree that can be updated, but not committed back to the main repository. To do this, use the following command: - &prompt.user; svn co https://svn0.us-west.FreeBSD.org/base/head /usr/src + &prompt.user; svn co https://svn.FreeBSD.org/base/head /usr/src - Select the closest mirror and verify the mirror server - certificate from the list of Subversion - mirror sites. + More details on using Subversion this way can be found in Using + Subversion. Updating the Tree To update a working copy to either the latest revision, or a specific revision: &prompt.user; svn update &prompt.user; svn update -r12345 Status To view the local changes that have been made to the working copy: &prompt.user; svn status To show local changes and files that are out-of-date do: &prompt.user; svn status --show-updates Editing and Committing Unlike Perforce, SVN does not need to be told in advance about file editing. To commit all changes in the current directory and all subdirectories: &prompt.user; svn commit To commit all changes in, for example, lib/libfetch/ and usr/bin/fetch/ in a single operation: &prompt.user; svn commit lib/libfetch usr/bin/fetch There is also a commit wrapper for the ports tree to handle the properties and sanity checking your changes: &prompt.user; /usr/ports/Tools/scripts/psvn commit Adding and Removing Files Before adding files, get a copy of auto-props.txt (there is also a ports tree specific version) and add it to ~/.subversion/config according to the instructions in the file. If you added something before reading this, use svn rm --keep-local for just added files, fix your config file and re-add them again. The initial config file is created when you first run a svn command, even something as simple as svn help. Files are added to a SVN repository with svn add. To add a file named foo, edit it, then: &prompt.user; svn add foo Most new source files should include a $&os;$ string near the start of the file. On commit, svn will expand the $&os;$ string, adding the file path, revision number, date and time of commit, and the username of the committer. Files which cannot be modified may be committed without the $&os;$ string. Files can be removed with svn remove: &prompt.user; svn remove foo Subversion does not require deleting the file before using svn rm, and indeed complains if that happens. It is possible to add directories with svn add: &prompt.user; mkdir bar &prompt.user; svn add bar Although svn mkdir makes this easier by combining the creation of the directory and the adding of it: &prompt.user; svn mkdir bar Like files, directories are removed with svn rm. There is no separate command specifically for removing directories. &prompt.user; svn rm bar Copying and Moving Files This command creates a copy of foo.c named bar.c, with the new file also under version control: &prompt.user; svn copy foo.c bar.c The example above is equivalent to: &prompt.user; cp foo.c bar.c &prompt.user; svn add bar.c To move and rename a file: &prompt.user; svn move foo.c bar.c Log and Annotate svn log shows revisions and commit messages, most recent first, for files or directories. When used on a directory, all revisions that affected the directory and files within that directory are shown. svn annotate, or equally svn praise or svn blame, shows the most recent revision number and who committed that revision for each line of a file. Diffs svn diff displays changes to the working copy. Diffs generated by SVN are unified and include new files by default in the diff output. svn diff can show the changes between two revisions of the same file: &prompt.user; svn diff -r179453:179454 ROADMAP.txt It can also show all changes for a specific changeset. The following will show what changes were made to the current directory and all subdirectories in changeset 179454: &prompt.user; svn diff -c179454 . Reverting Local changes (including additions and deletions) can be reverted using svn revert. It does not update out-of-date files, but just replaces them with pristine copies of the original version. Conflicts If an svn update resulted in a merge conflict, Subversion will remember which files have conflicts and refuse to commit any changes to those files until explicitly told that the conflicts have been resolved. The simple, not yet deprecated procedure is the following: &prompt.user; svn resolved foo However, the preferred procedure is: &prompt.user; svn resolve --accept=working foo The two examples are equivalent. Possible values for --accept are: working: use the version in your working directory (which one presumes has been edited to resolve the conflicts). base: use a pristine copy of the version you had before svn update, discarding your own changes, the conflicting changes, and possibly other intervening changes as well. mine-full: use what you had before svn update, including your own changes, but discarding the conflicting changes, and possibly other intervening changes as well. theirs-full: use the version that was retrieved when you did svn update, discarding your own changes. Advanced Use Sparse Checkouts SVN allows sparse, or partial checkouts of a directory by adding to a svn checkout. Valid arguments to are: empty: the directory itself without any of its contents. files: the directory and any files it contains. immediates: the directory and any files and directories it contains, but none of the subdirectories' contents. infinity: anything. The --depth option applies to many other commands, including svn commit, svn revert, and svn diff. Since --depth is sticky, there is a --set-depth option for svn update that will change the selected depth. Thus, given the working copy produced by the previous example: &prompt.user; cd ~/freebsd &prompt.user; svn update --set-depth=immediates . The above command will populate the working copy in ~/freebsd with ROADMAP.txt and empty subdirectories, and nothing will happen when svn update is executed on the subdirectories. However, the following command will set the depth for head (in this case) to infinity, and fully populate it: &prompt.user; svn update --set-depth=infinity head Direct Operation Certain operations can be performed directly on the repository without touching the working copy. Specifically, this applies to any operation that does not require editing a file, including: log, diff mkdir remove, copy, rename propset, propedit, propdel merge Branching is very fast. The following command would be used to branch RELENG_8: &prompt.user; svn copy svn+ssh://repo.freebsd.org/base/head svn+ssh://repo.freebsd.org/base/stable/8 This is equivalent to the following set of commands which take minutes and hours as opposed to seconds, depending on your network connection: &prompt.user; svn checkout --depth=immediates svn+ssh://repo.freebsd.org/base &prompt.user; cd base &prompt.user; svn update --set-depth=infinity head &prompt.user; svn copy head stable/8 &prompt.user; svn commit stable/8 Merging with <acronym>SVN</acronym> This section deals with merging code from one branch to another (typically, from head to a stable branch). In all examples below, $FSVN refers to the location of the &os; Subversion repository, svn+ssh://repo.freebsd.org/base/. About Merge Tracking From the user's perspective, merge tracking information (or mergeinfo) is stored in a property called svn:mergeinfo, which is a comma-separated list of revisions and ranges of revisions that have been merged. When set on a file, it applies only to that file. When set on a directory, it applies to that directory and its descendants (files and directories) except for those that have their own svn:mergeinfo. It is not inherited. For instance, stable/6/contrib/openpam/ does not implicitly inherit mergeinfo from stable/6/, or stable/6/contrib/. Doing so would make partial checkouts very hard to manage. Instead, mergeinfo is explicitly propagated down the tree. For merging something into branch/foo/bar/, the following rules apply: If branch/foo/bar/ does not already have a mergeinfo record, but a direct ancestor (for instance, branch/foo/) does, then that record will be propagated down to branch/foo/bar/ before information about the current merge is recorded. Information about the current merge will not be propagated back up that ancestor. If a direct descendant of branch/foo/bar/ (for instance, branch/foo/bar/baz/) already has a mergeinfo record, information about the current merge will be propagated down to it. If you consider the case where a revision changes several separate parts of the tree (for example, branch/foo/bar/ and branch/foo/quux/), but you only want to merge some of it (for example, branch/foo/bar/), you will see that these rules make sense. If mergeinfo was propagated up, it would seem like that revision had also been merged to branch/foo/quux/, when in fact it had not been. Selecting the Source and Target for <literal>stable/10</literal> and Newer Starting with the stable/10 branch, all merges should be merged to and committed from the root of the branch. All merges should look like: &prompt.user; svn merge -c r123456 ^/head/ checkout &prompt.user; svn commit checkout Note that checkout should be a complete checkout of the branch to which the merge occurs. Merges to releng/ branches should always originate from the corresponding stable/ branch. For example: &prompt.user; svn merge -c r123456 ^/stable/10 releng/10.0 Selecting the Source and Target for <literal>stable/9</literal> and Older For stable/9 and earlier, a different strategy was used, distributing mergeinfo around the tree so that merges could be performed without a complete checkout. This procedure proved extremely error-prone, with the convenience of partial checkouts for merges significantly outweighed by the complexity of picking mergeinfo targets. The below describes this now-obsoleted procedure, which should be used only for merges prior to stable/10. Because of mergeinfo propagation, it is important to choose the source and target for the merge carefully to minimise property changes on unrelated directories. The rules for selecting the merge target (the directory that you will merge the changes to) can be summarized as follows: Never merge directly to a file. Never, ever merge directly to a file. Never, ever, ever merge directly to a file. Changes to kernel code should be merged to sys/. For instance, a change to the &man.ichwd.4; driver should be merged to sys/, not sys/dev/ichwd/. Likewise, a change to the TCP/IP stack should be merged to sys/, not sys/netinet/. Changes to code under etc/ should be merged at etc/, not below it. Changes to vendor code (code in contrib/, crypto/ and so on) should be merged to the directory where vendor imports happen. For instance, a change to crypto/openssl/util/ should be merged to crypto/openssl/. This is rarely an issue, however, since changes to vendor code are usually merged wholesale. Changes to userland programs should as a general rule be merged to the directory that contains the Makefile for that program. For instance, a change to usr.bin/xlint/arch/i386/ should be merged to usr.bin/xlint/. Changes to userland libraries should as a general rule be merged to the directory that contains the Makefile for that library. For instance, a change to lib/libc/gen/ should be merged to lib/libc/. There may be cases where it makes sense to deviate from the rules for userland programs and libraries. For instance, everything under lib/libpam/ is merged to lib/libpam/, even though the library itself and all of the modules each have their own Makefile. Changes to manual pages should be merged to share/man/manN/, for the appropriate value of N. Other changes to share/ should be merged to the appropriate subdirectory and not to share/ directly. Changes to a top-level file in the source tree such as UPDATING or Makefile.inc1 should be merged directly to that file rather than to the root of the whole tree. Yes, this is an exception to the first three rules. When in doubt, ask. If you need to merge changes to several places at once (for instance, changing a kernel interface and every userland program that uses it), merge each target separately, then commit them together. For instance, if you merge a revision that changed a kernel API and updated all the userland bits that used that API, you would merge the kernel change to sys, and the userland bits to the appropriate userland directories, then commit all of these in one go. The source will almost invariably be the same as the target. For instance, you will always merge stable/7/lib/libc/ from head/lib/libc/. The only exception would be when merging changes to code that has moved in the source branch but not in the parent branch. For instance, a change to &man.pkill.1; would be merged from bin/pkill/ in head to usr.bin/pkill/ in stable/7. Preparing the Merge Target Because of the mergeinfo propagation issues described earlier, it is very important that you never merge changes into a sparse working copy. You must always have a full checkout of the branch you will merge into. For instance, when merging from HEAD to 7, you must have a full checkout of stable/7: &prompt.user; cd stable/7 &prompt.user; svn up --set-depth=infinity The target directory must also be up-to-date and must not contain any uncommitted changes or stray files. Identifying Revisions Identifying revisions to be merged is a must. If the target already has complete mergeinfo, ask SVN for a list: &prompt.user; cd stable/6/contrib/openpam &prompt.user; svn mergeinfo --show-revs=eligible $FSVN/head/contrib/openpam If the target does not have complete mergeinfo, check the log for the merge source. Merging Now, let us start merging! The Principles Say you would like to merge: revision $R in directory $target in stable branch $B from directory $source in head $FSVN is svn+ssh://repo.freebsd.org/base Assuming that revisions $P and $Q have already been merged, and that the current directory is an up-to-date working copy of stable/$B, the existing mergeinfo looks like this: &prompt.user; svn propget svn:mergeinfo -R $target $target - /head/$source:$P,$Q Merging is done like so: &prompt.user; svn merge -c$R $FSVN/head/$source $target Checking the results of this is possible with svn diff. The svn:mergeinfo now looks like: &prompt.user; svn propget svn:mergeinfo -R $target $target - head/$source:$P,$Q,$R If the results are not exactly as shown, assistance may be required before committing as mistakes may have been made, or there may be something wrong with the existing mergeinfo, or there may be a bug in Subversion. Practical Example As a practical example, consider the following scenario. The changes to netmap.4 in r238987 are to be merged from CURRENT to 9-STABLE. The file resides in head/share/man/man4. According to , this is also where to do the merge. Note that in this example all paths are relative to the top of the svn repository. For more information on the directory layout, see . The first step is to inspect the existing mergeinfo. &prompt.user; svn propget svn:mergeinfo -R stable/9/share/man/man4 Take a quick note of how it looks before moving on to the next step; doing the actual merge: &prompt.user; svn merge -c r238987 svn+ssh://repo.freebsd.org/base/head/share/man/man4 stable/9/share/man/man4 --- Merging r238987 into 'stable/9/share/man/man4': U stable/9/share/man/man4/netmap.4 --- Recording mergeinfo for merge of r238987 into 'stable/9/share/man/man4': U stable/9/share/man/man4 Check that the revision number of the merged revision has been added. Once this is verified, the only thing left is the actual commit. &prompt.user; svn commit stable/9/share/man/man4 Merging into the Kernel (<filename>sys/</filename>) As stated above, merging into the kernel is different from merging in the rest of the tree. In many ways merging to the kernel is simpler because there is always the same merge target (sys/). Once svn merge has been executed, svn diff has to be run on the directory to check the changes. This may show some unrelated property changes, but these can be ignored. Next, build and test the kernel, and, once the tests are complete, commit the code as normal, making sure that the commit message starts with Merge r226222 from head, or similar. Precautions Before Committing As always, build world (or appropriate parts of it). Check the changes with svn diff and svn stat. Make sure all the files that should have been added or deleted were in fact added or deleted. Take a closer look at any property change (marked by a M in the second column of svn stat). Normally, no svn:mergeinfo properties should be anywhere except the target directory (or directories). If something looks fishy, ask for help. Committing Make sure to commit a top level directory to have the mergeinfo included as well. Do not specify individual files on the command line. For more information about committing files in general, see the relevant section of this primer. Vendor Imports with <acronym>SVN</acronym> Please read this entire section before starting a vendor import. Patches to vendor code fall into two categories: Vendor patches: these are patches that have been issued by the vendor, or that have been extracted from the vendor's version control system, which address issues which in your opinion cannot wait until the next vendor release. &os; patches: these are patches that modify the vendor code to address &os;-specific issues. The nature of a patch dictates where it should be committed: Vendor patches should be committed to the vendor branch, and merged from there to head. If the patch addresses an issue in a new release that is currently being imported, it must not be committed along with the new release: the release must be imported and tagged first, then the patch can be applied and committed. There is no need to re-tag the vendor sources after committing the patch. &os; patches should be committed directly to head. Preparing the Tree If importing for the first time after the switch to Subversion, flattening and cleaning up the vendor tree is necessary, as well as bootstrapping the merge history in the main tree. Flattening During the conversion from CVS to Subversion, vendor branches were imported with the same layout as the main tree. This means that the pf vendor sources ended up in vendor/pf/dist/contrib/pf. The vendor source is best directly in vendor/pf/dist. To flatten the pf tree: &prompt.user; cd vendor/pf/dist/contrib/pf &prompt.user; svn mv $(svn list) ../.. &prompt.user; cd ../.. &prompt.user; svn rm contrib &prompt.user; svn propdel -R svn:mergeinfo . &prompt.user; svn commit The propdel bit is necessary because starting with 1.5, Subversion will automatically add svn:mergeinfo to any directory that is copied or moved. In this case, as nothing is being merged from the deleted tree, they just get in the way. Tags may be flattened as well (3, 4, 3.5 etc.); the procedure is exactly the same, only changing dist to 3.5 or similar, and putting the svn commit off until the end of the process. Cleaning Up The dist tree can be cleaned up as necessary. Disabling keyword expansion is recommended, as it makes no sense on unmodified vendor code and in some cases it can even be harmful. OpenSSH, for example, includes two files that originated with &os; and still contain the original version tags. To do this: &prompt.user; svn propdel svn:keywords -R . &prompt.user; svn commit Bootstrapping Merge History If importing for the first time after the switch to Subversion, bootstrap svn:mergeinfo on the target directory in the main tree to the revision that corresponds to the last related change to the vendor tree, prior to importing new sources: &prompt.user; cd head/contrib/pf &prompt.user; svn merge --record-only svn+ssh://repo.freebsd.org/base/vendor/pf/dist@180876 . &prompt.user; svn commit Importing New Sources With two commits—one for the import itself and one for the tag—this step can optionally be repeated for every upstream release between the last import and the current import. Preparing the Vendor Sources Unlike in CVS where only the needed parts were imported into the vendor tree to avoid bloating the main tree, Subversion is able to store a full distribution in the vendor tree. So, import everything, but merge only what is required. A svn add is required to add any files that were added since the last vendor import, and svn rm is required to remove any that were removed since. Preparing sorted lists of the contents of the vendor tree and of the sources that are about to be imported is recommended, to facilitate the process. &prompt.user; cd vendor/pf/dist &prompt.user; svn list -R | grep -v '/$' | sort >../old &prompt.user; cd ../pf-4.3 &prompt.user; find . -type f | cut -c 3- | sort >../new With these two files, comm -23 ../old ../new will list removed files (files only in old), while comm -13 ../old ../new will list added files only in new. Importing into the Vendor Tree Now, the sources must be copied into dist and the svn add and svn rm commands should be used as needed: &prompt.user; cd vendor/pf/pf-4.3 &prompt.user; tar cf - . | tar xf - -C ../dist &prompt.user; cd ../dist &prompt.user; comm -23 ../old ../new | xargs svn rm &prompt.user; comm -13 ../old ../new | xargs svn --parents add If any directories were removed, they will have to be svn rmed manually. Nothing will break if they are not, but they will remain in the tree. Check properties on any new files. All text files should have svn:eol-style set to native. All binary files should have svn:mime-type set to application/octet-stream unless there is a more appropriate media type. Executable files should have svn:executable set to *. No other properties should exist on any file in the tree. Committing is now possible, however it is good practice to make sure that everything is OK by using the svn stat and svn diff commands. Tagging Once committed, vendor releases should be tagged for future reference. The best and quickest way to do this is directly in the repository: &prompt.user; svn cp svn+ssh://repo.freebsd.org/base/vendor/pf/dist svn+ssh://repo.freebsd.org/base/vendor/pf/4.3 Once that is complete, svn up the working copy of vendor/pf to get the new tag, although this is rarely needed. If creating the tag in the working copy of the tree, svn:mergeinfo results must be removed: &prompt.user; cd vendor/pf &prompt.user; svn cp dist 4.3 &prompt.user; svn propdel svn:mergeinfo -R 4.3 Merging to Head &prompt.user; cd head/contrib/pf &prompt.user; svn up &prompt.user; svn merge --accept=postpone svn+ssh://repo.freebsd.org/base/vendor/pf/dist . The --accept=postpone tells Subversion that it should not complain because merge conflicts will be taken care of manually. The cvs2svn changeover occurred on June 3, 2008. When performing vendor merges for packages which were already present and converted by the cvs2svn process, the command used to merge /vendor/package_name/dist to /head/package_location (for example, head/contrib/sendmail) must use to indicate the revision to merge from the /vendor tree. For example: &prompt.user; svn checkout svn+ssh://repo.freebsd.org/base/head/contrib/sendmail &prompt.user; cd sendmail &prompt.user; svn merge -c r261190 ^/vendor/sendmail/dist . ^ is an alias for the repository path. If using the Zsh shell, the ^ must be escaped with \. This means ^/head should be \^/head. It is necessary to resolve any merge conflicts. Make sure that any files that were added or removed in the vendor tree have been properly added or removed in the main tree. To check diffs against the vendor branch: &prompt.user; svn diff --no-diff-deleted --old=svn+ssh://repo.freebsd.org/base/vendor/pf/dist --new=. The --no-diff-deleted tells Subversion not to complain about files that are in the vendor tree but not in the main tree, i.e., things that would have previously been removed before the vendor import, like for example the vendor's makefiles and configure scripts. Using CVS, once a file was off the vendor branch, it was not able to be put back. With Subversion, there is no concept of on or off the vendor branch. If a file that previously had local modifications, to make it not show up in diffs in the vendor tree, all that has to be done is remove any left-over cruft like &os; version tags, which is much easier. If any changes are required for the world to build with the new sources, make them now, and keep testing until everything builds and runs perfectly. Committing the Vendor Import Committing is now possible! Everything must be committed in one go. If done properly, the tree will move from a consistent state with old code, to a consistent state with new code. From Scratch Importing into the Vendor Tree This section is an example of importing and tagging byacc into head. First, prepare the directory in vendor: &prompt.user; svn co --depth immediates $FSVN/vendor &prompt.user; cd vendor &prompt.user; svn mkdir byacc &prompt.user; svn mkdir byacc/dist Now, import the sources into the dist directory. Once the files are in place, svn add the new ones, then svn commit and tag the imported version. To save time and bandwidth, direct remote committing and tagging is possible: &prompt.user; svn cp -m "Tag byacc 20120115" $FSVN/vendor/byacc/dist $FSVN/vendor/byacc/20120115 Merging to <literal>head</literal> Due to this being a new file, copy it for the merge: &prompt.user; svn cp -m "Import byacc to contrib" $FSVN/vendor/byacc/dist $FSVN/head/contrib/byacc Working normally on newly imported sources is still possible. Reverting a Commit Reverting a commit to a previous version is fairly easy: &prompt.user; svn merge -r179454:179453 ROADMAP.txt &prompt.user; svn commit Change number syntax, with negative meaning a reverse change, can also be used: &prompt.user; svn merge -c -179454 ROADMAP.txt &prompt.user; svn commit This can also be done directly in the repository: &prompt.user; svn merge -r179454:179453 svn+ssh://repo.freebsd.org/base/ROADMAP.txt It is important to ensure that the mergeinfo is correct when reverting a file in order to permit svn mergeinfo --eligible to work as expected. Reverting the deletion of a file is slightly different. Copying the version of the file that predates the deletion is required. For example, to restore a file that was deleted in revision N, restore version N-1: &prompt.user; svn copy svn+ssh://repo.freebsd.org/base/ROADMAP.txt@179454 &prompt.user; svn commit or, equally: &prompt.user; svn copy svn+ssh://repo.freebsd.org/base/ROADMAP.txt@179454 svn+ssh://repo.freebsd.org/base Do not simply recreate the file manually and svn add it—this will cause history to be lost. Fixing Mistakes While we can do surgery in an emergency, do not plan on having mistakes fixed behind the scenes. Plan on mistakes remaining in the logs forever. Be sure to check the output of svn status and svn diff before committing. Mistakes will happen but, they can generally be fixed without disruption. Take a case of adding a file in the wrong location. The right thing to do is to svn move the file to the correct location and commit. This causes just a couple of lines of metadata in the repository journal, and the logs are all linked up correctly. The wrong thing to do is to delete the file and then svn add an independent copy in the correct location. Instead of a couple of lines of text, the repository journal grows an entire new copy of the file. This is a waste. Setting up a <application>svnsync</application> Mirror You probably do not want to do this unless there is a good reason for it. Such reasons might be to support many multiple local read-only client machines, or if your network bandwidth is limited. Starting a fresh mirror from empty would take a very long time. Expect a minimum of 10 hours for high speed connectivity. If you have international links, expect this to take 4 to 10 times longer. A far better option is to grab a seed file. It is large (~1GB) but will consume less network traffic and take less time to fetch than a svnsync will. This is possible in one of the following three ways: &prompt.user; rsync -va --partial --progress freefall:/home/peter/svnmirror-base-r179637.tbz2 . &prompt.user; rsync -va --partial --progress rsync://repoman.freebsd.org:50873/svnseed/svnmirror-base-r215629.tar.xz . &prompt.user; fetch ftp://ftp.freebsd.org/pub/FreeBSD/development/subversion/svnmirror-base-r221445.tar.xz Once you have the file, extract it to somewhere like home/svnmirror/base/. Then, update it, so that it fetches changes since the last revision in the archive: &prompt.user; svnsync sync file:///home/svnmirror/base You can then set that up to run from &man.cron.8;, do checkouts locally, set up a svnserve server for your local machines to talk to, etc. The seed mirror is set to fetch from svn://svn.freebsd.org/base. The configuration for the mirror is stored in revprop 0 on the local mirror. To see the configuration, try: &prompt.user; svn proplist -v --revprop -r 0 file:///home/svnmirror/base Use propset to change things. Committing High-<acronym>ASCII</acronym> Data Files that have high-ASCII bits are considered binary files in SVN, so the pre-commit checks fail and indicate that the mime-type property should be set to application/octet-stream. However, the use of this is discouraged, so please do not set it. The best way is always avoiding high-ASCII data, so that it can be read everywhere with any text editor but if it is not avoidable, instead of changing the mime-type, set the fbsd:notbinary property with propset: &prompt.user; svn propset fbsd:notbinary yes foo.data Maintaining a Project Branch A project branch is one that is synced to head (or another branch) is used to develop a project then commit it back to head. In SVN, dolphin branching is used for this. A dolphin branch is one that diverges for a while and is finally committed back to the original branch. During development code migration in one direction (from head to the branch only). No code is committed back to head until the end. Once you commit back at the end, the branch is dead (although you can have a new branch with the same name after you delete the branch if you want). As per http://people.freebsd.org/~peter/svn_notes.txt, work that is intended to be merged back into HEAD should be in base/projects/. If you are doing work that is beneficial to the &os; community in some way but not intended to be merged directly back into HEAD then the proper location is base/user/your-name/. This page contains further details. To create a project branch: &prompt.user; svn copy svn+ssh://repo.freebsd.org/base/head svn+ssh://repo.freebsd.org/base/projects/spif To merge changes from HEAD back into the project branch: &prompt.user; cd copy_of_spif &prompt.user; svn merge svn+ssh://repo.freebsd.org/base/head &prompt.user; svn commit It is important to resolve any merge conflicts before committing. Some Tips In commit logs etc., rev 179872 should be spelled r179872 as per convention. Speeding up svn is possible by adding the following to ~/.ssh/config: Host * ControlPath ~/.ssh/sockets/master-%l-%r@%h:%p ControlMaster auto ControlPersist yes and then typing mkdir ~/.ssh/sockets Checking out a working copy with a stock Subversion client without &os;-specific patches (OPTIONS_SET=FREEBSD_TEMPLATE) will mean that $FreeBSD$ tags will not be expanded. Once the correct version has been installed, trick Subversion into expanding them like so: &prompt.user; svn propdel -R svn:keywords . &prompt.user; svn revert -R . This will wipe out uncommitted patches. It is possible to automatically fill the "Sponsored by" and "MFC after" commit log fields by setting "freebsd-sponsored-by" and "freebsd-mfc-after" fields in the "[miscellany]" section of the ~/.subversion/config configuration file. For example: freebsd-sponsored-by = The FreeBSD Foundation freebsd-mfc-after = 2 weeks Setup, Conventions, and Traditions There are a number of things to do as a new developer. The first set of steps is specific to committers only. These steps must be done by a mentor for those who are not committers. For New Committers Those who have been given commit rights to the &os; repositories must follow these steps. Get mentor approval before committing each of these changes! The .ent and .xml files mentioned below exist in the &os; Documentation Project SVN repository at svn.FreeBSD.org/doc/. New files that do not have the FreeBSD=%H svn:keywords property will be rejected when attempting to commit them to the repository. Be sure to read regarding adding and removing files. Verify that ~/.subversion/config contains the necessary auto-props entries from auto-props.txt mentioned there. All src commits should go to &os.current; first before being merged to &os.stable;. The &os.stable; branch must maintain ABI and API compatibility with earlier versions of that branch. Do not merge changes that break this compatibility. Steps for New Committers Add an Author Entity doc/head/share/xml/authors.ent — Add an author entity. Later steps depend on this entity, and missing this step will cause the doc/ build to fail. This is a relatively easy task, but remains a good first test of version control skills. Update the List of Developers and Contributors doc/head/en_US.ISO8859-1/articles/contributors/contrib.committers.xml — Add an entry to the Developers section of the Contributors List. Entries are sorted by last name. doc/head/en_US.ISO8859-1/articles/contributors/contrib.additional.xml — Remove the entry from the Additional Contributors section. Entries are sorted by last name. Add a News Item doc/head/share/xml/news.xml — Add an entry. Look for the other entries that announce new committers and follow the format. Use the date from the commit bit approval email from core@FreeBSD.org. Add a <acronym>PGP</acronym> Key doc/head/share/pgpkeys/pgpkeys.ent and doc/head/share/pgpkeys/pgpkeys-developers.xml - Add your PGP or GnuPG key. Those who do not yet have a key should see . &a.des.email; has written a shell script (doc/head/share/pgpkeys/addkey.sh) to make this easier. See the README file for more information. Use doc/head/share/pgpkeys/checkkey.sh to verify that keys meet minimal best-practices standards. After adding and checking a key, add both updated files to source control and then commit them. Entries in this file are sorted by last name. It is very important to have a current PGP/GnuPG key in the repository. The key may be required for positive identification of a committer. For example, the &a.admins; might need it for account recovery. A complete keyring of FreeBSD.org users is available for download from http://www.FreeBSD.org/doc/pgpkeyring.txt. Update Mentor and Mentee Information base/head/share/misc/committers-repository.dot — Add an entry to the current committers section, where repository is doc, ports, or src, depending on the commit privileges granted. Add an entry for each additional mentor/mentee relationship in the bottom section. Generate a <application>Kerberos</application> Password See to generate or set a Kerberos for use with other &os; services like the bug tracking database. Optional: Enable Wiki Account &os; Wiki Account — A wiki account allows sharing projects and ideas. Those who do not yet have an account can contact clusteradm@FreeBSD.org to obtain one. Optional: Update Wiki Information Wiki Information - After gaining access to the wiki, some people add entries to the How We Got Here, Irc Nicks, and Dogs of FreeBSD pages. Optional: Update Ports with Personal Information ports/astro/xearth/files/freebsd.committers.markers and src/usr.bin/calendar/calendars/calendar.freebsd - Some people add entries for themselves to these files to show where they are located or the date of their birthday. Optional: Prevent Duplicate Mailings Subscribers to &a.svn-src-all.name;, &a.svn-ports-all.name; or &a.svn-doc-all.name; might wish to unsubscribe to avoid receiving duplicate copies of commit messages and followups. For Everyone Introduce yourself to the other developers, otherwise no one will have any idea who you are or what you are working on. The introduction need not be a comprehensive biography, just write a paragraph or two about who you are, what you plan to be working on as a developer in &os;, and who will be your mentor. Email this to the &a.developers; and you will be on your way! Log into freefall.FreeBSD.org and create a /var/forward/user (where user is your username) file containing the e-mail address where you want mail addressed to yourusername@FreeBSD.org to be forwarded. This includes all of the commit messages as well as any other mail addressed to the &a.committers; and the &a.developers;. Really large mailboxes which have taken up permanent residence on freefall may get truncated without warning if space needs to be freed, so forward it or read it and you will not lose it. Due to the severe load dealing with SPAM places on the central mail servers that do the mailing list processing the front-end server does do some basic checks and will drop some messages based on these checks. At the moment proper DNS information for the connecting host is the only check in place but that may change. Some people blame these checks for bouncing valid email. If you want these checks turned off for your email you can place a file named .spam_lover in your home directory on freefall.FreeBSD.org to disable the checks for your email. Those who are developers but not committers will not be subscribed to the committers or developers mailing lists. The subscriptions are derived from the access rights. Mentors All new developers have a mentor assigned to them for the first few months. A mentor is responsible for teaching the mentee the rules and conventions of the project and guiding their first steps in the developer community. The mentor is also personally responsible for the mentee's actions during this initial period. For committers: do not commit anything without first getting mentor approval. Document that approval with an Approved by: line in the commit message. When the mentor decides that a mentee has learned the ropes and is ready to commit on their own, the mentor announces it with a commit to mentors. Commit Log Messages This section contains some suggestions and traditions for how commit logs are formatted. As well as including an informative message with each commit you may need to include some additional information. This information consists of one or more lines containing the key word or phrase, a colon, tabs for formatting, and then the additional information. The key words or phrases are: PR: The problem report (if any) which is affected (typically, by being closed) by this commit. Only include one PR per line as the automated scripts which parse this line cannot understand more than one. Differential Revision: The full URL of the Phabricator review. For example: https://reviews.freebsd.org/D1708. Submitted by: The name and e-mail address of the person that submitted the fix; for developers, just the username on the &os; cluster. If the submitter is the maintainer of the port to which you are committing, include "(maintainer)" after the email address. Avoid obfuscating the email address of the submitter as this adds additional work when searching logs. Reviewed by: The name and e-mail address of the person or people that reviewed the change; for developers, just the username on the &os; cluster. If a patch was submitted to a mailing list for review, and the review was favorable, then just include the list name. Approved by: The name and e-mail address of the person or people that approved the change; for developers, just the username on the &os; cluster. It is customary to get prior approval for a commit if it is to an area of the tree to which you do not usually commit. In addition, during the run up to a new release all commits must be approved by the release engineering team. While under mentorship, get mentor approval before the commit. Enter the mentor's username in this field, and note that they are a mentor: Approved by: username-of-mentor (mentor) If a team approved these commits then include the team name followed by the username of the approver in parentheses. For example: Approved by: re (username) Obtained from: The name of the project (if any) from which the code was obtained. Do not use this line for the name of an individual person. MFC after: If you wish to receive an e-mail reminder to MFC at a later date, specify the number of days, weeks, or months after which an MFC is planned. Relnotes: If the change is a candidate for inclusion in the release notes for the next release from the branch, set to yes. Security: If the change is related to a security vulnerability or security exposure, include one or more references or a description of the issue. If possible, include a VuXML URL or a CVE ID. Commit Log for a Commit Based on a PR You want to commit a change based on a PR submitted by John Smith containing a patch. The end of the commit message should look something like this. ... PR: 12345 Submitted by: John Smith <John.Smith@example.com> Commit Log for a Commit Needing Review You want to change the virtual memory system. You have posted patches to the appropriate mailing list (in this case, freebsd-arch) and the changes have been approved. ... Reviewed by: -arch Commit Log for a Commit Needing Approval You want to commit a port. You have collaborated with the listed MAINTAINER, who has told you to go ahead and commit. ... Approved by: abc (maintainer) Where abc is the account name of the person who approved. Commit Log for a Commit Bringing in Code from OpenBSD You want to commit some code based on work done in the OpenBSD project. ... Obtained from: OpenBSD Commit Log for a Change to &os.current; with a Planned Commit to &os.stable; to Follow at a Later Date. You want to commit some code which will be merged from &os.current; into the &os.stable; branch after two weeks. ... MFC after: 2 weeks Where 2 is the number of days, weeks, or months after which an MFC is planned. The weeks option may be day, days, week, weeks, month, months. In many cases you may need to combine some of these. Consider the situation where a user has submitted a PR containing code from the NetBSD project. You are looking at the PR, but it is not an area of the tree you normally work in, so you have decided to get the change reviewed by the arch mailing list. Since the change is complex, you opt to MFC after one month to allow adequate testing. The extra information to include in the commit would look something like Example Combined Commit Log PR: 54321 Submitted by: John Smith <John.Smith@example.com> Reviewed by: -arch Obtained from: NetBSD MFC after: 1 month Relnotes: yes Preferred License for New Files Currently the &os; Project suggests and uses the following text as the preferred license scheme: /*- * Copyright (c) [year] [your name] * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * [id for your version control system, if any] */ The &os; project strongly discourages the so-called "advertising clause" in new code. Due to the large number of contributors to the &os; project, complying with this clause for many commercial vendors has become difficult. If you have code in the tree with the advertising clause, please consider removing it. In fact, please consider using the above license for your code. The &os; project discourages completely new licenses and variations on the standard licenses. New licenses require the approval of the &a.core; to reside in the main repository. The more different licenses that are used in the tree, the more problems that this causes to those wishing to utilize this code, typically from unintended consequences from a poorly worded license. Project policy dictates that code under some non-BSD licenses must be placed only in specific sections of the repository, and in some cases, compilation must be conditional or even disabled by default. For example, the GENERIC kernel must be compiled under only licenses identical to or substantially similar to the BSD license. GPL, APSL, CDDL, etc, licensed software must not be compiled into GENERIC. Developers are reminded that in open source, getting "open" right is just as important as getting "source" right, as improper handling of intellectual property has serious consequences. Any questions or concerns should immediately be brought to the attention of the core team. Keeping Track of Licenses Granted to the &os; Project Various software or data exist in the repositories where the &os; project has been granted a special licence to be able to use them. A case in point are the Terminus fonts for use with &man.vt.4;. Here the author Dimitar Zhekov has allowed us to use the "Terminus BSD Console" font under a 2-clause BSD license rather than the regular Open Font License he normally uses. It is clearly sensible to keep a record of any such license grants. To that end, the &a.core; has decided to keep an archive of them. Whenever the &os; project is granted a special license we require the &a.core; to be notified. Any developers involved in arranging such a license grant, please send details to the &a.core; including: Contact details for people or organizations granting the special license. What files, directories etc. in the repositories are covered by the license grant including the revision numbers where any specially licensed material was committed. The date the license comes into effect from. Unless otherwise agreed, this will be the date the license was issued by the authors of the software in question. The license text. A note of any restrictions, limitations or exceptions that apply specifically to &os;'s usage of the licensed material. Any other relevant information. Once the &a.core; is satisfied that all the necessary details have been gathered and are correct, the secretary will send a PGP-signed acknowledgement of receipt including the license details. This receipt will be persistently archived and serve as our permanent record of the license grant. The license archive should contain only details of license grants; this is not the place for any discussions around licensing or other subjects. Access to data within the license archive will be available on request to the &a.core;. Developer Relations If you are working directly on your own code or on code which is already well established as your responsibility, then there is probably little need to check with other committers before jumping in with a commit. If you see a bug in an area of the system which is clearly orphaned (and there are a few such areas, to our shame), the same applies. If, however, you are about to modify something which is clearly being actively maintained by someone else (and it is only by watching the repository-committers mailing list that you can really get a feel for just what is and is not) then consider sending the change to them instead, just as you would have before becoming a committer. For ports, you should contact the listed MAINTAINER in the Makefile. For other parts of the repository, if you are unsure who the active maintainer might be, it may help to scan the revision history to see who has committed changes in the past. An example script that lists each person who has committed to a given file along with the number of commits each person has made can be found at on freefall at ~eadler/bin/whodid. If your queries go unanswered or the committer otherwise indicates a lack of interest in the area affected, go ahead and commit it. Avoid sending private emails to maintainers. Other people might be interested in the conversation, not just the final output. If you are unsure about a commit for any reason at all, have it reviewed by -hackers before committing. Better to have it flamed then and there rather than when it is part of the repository. If you do happen to commit something which results in controversy erupting, you may also wish to consider backing the change out again until the matter is settled. Remember – with a version control system we can always change it back. Do not impugn the intentions of someone you disagree with. If they see a different solution to a problem than you, or even a different problem, it is not because they are stupid, because they have questionable parentage, or because they are trying to destroy your hard work, personal image, or &os;, but simply because they have a different outlook on the world. Different is good. Disagree honestly. Argue your position from its merits, be honest about any shortcomings it may have, and be open to seeing their solution, or even their vision of the problem, with an open mind. Accept correction. We are all fallible. When you have made a mistake, apologize and get on with life. Do not beat up yourself, and certainly do not beat up others for your mistake. Do not waste time on embarrassment or recrimination, just fix the problem and move on. Ask for help. Seek out (and give) peer reviews. One of the ways open source software is supposed to excel is in the number of eyeballs applied to it; this does not apply if nobody will review code. If in Doubt... When you are not sure about something, whether it be a technical issue or a project convention be sure to ask. If you stay silent you will never make progress. If it relates to a technical issue ask on the public mailing lists. Avoid the temptation to email the individual person that knows the answer. This way everyone will be able to learn from the question and the answer. For project specific or administrative questions you should ask, in order: Your mentor or former mentor. An experienced committer on IRC, email, etc. Any team with a "hat", as they should give you a definitive answer. If still not sure, ask on &a.developers;. Once your question is answered, if no one pointed you to documentation that spelled out the answer to your question, document it, as others will have the same question. Bugzilla The &os; Project utilizes Bugzilla for tracking bugs and change requests. Be sure that if you commit a fix or suggestion found in the PR database to close it. It is also considered nice if you take time to close any PRs associated with your commits, if appropriate. Committers with non-&os;.org Bugzilla accounts can have the old account merged with the &os;.org account by entering a new bug. Choose Supporting Services as the Product, and Bug Tracker as the Component. You can find out more about Bugzilla at: &os; Problem Report Handling Guidelines http://www.FreeBSD.org/support.html Phabricator The &os; Project utilizes Phabricator for code review requests. See the CodeReview wiki page for details. Who's Who Besides the repository meisters, there are other &os; project members and teams whom you will probably get to know in your role as a committer. Briefly, and by no means all-inclusively, these are: &a.doceng; doceng is the group responsible for the documentation build infrastructure, approving new documentation committers, and ensuring that the &os; website and documentation on the FTP site is up to date with respect to the subversion tree. It is not a conflict resolution body. The vast majority of documentation related discussion takes place on the &a.doc;. More details regarding the doceng team can be found in its charter. Committers interested in contributing to the documentation should familiarize themselves with the Documentation Project Primer. &a.bde.email; Bruce is the Style Police-Meister. When you do a commit that could have been done better, Bruce will be there to tell you. Be thankful that someone is. Bruce is also very knowledgeable on the various standards applicable to &os;. &a.re.members.email; These are the members of the &a.re;. This team is responsible for setting release deadlines and controlling the release process. During code freezes, the release engineers have final authority on all changes to the system for whichever branch is pending release status. If there is something you want merged from &os.current; to &os.stable; (whatever values those may have at any given time), these are the people to talk to about it. Hiroki is also the keeper of the release documentation (src/release/doc/*). If you commit a change that you think is worthy of mention in the release notes, please make sure he knows about it. Better still, send him a patch with your suggested commentary. &a.des.email; Dag-Erling is the &os; Security Officer and oversees the &a.security-officer;. &a.wollman.email; If you need advice on obscure network internals or are not sure of some potential change to the networking subsystem you have in mind, Garrett is someone to talk to. Garrett is also very knowledgeable on the various standards applicable to &os;. &a.committers; &a.svn-src-all.name;, &a.svn-ports-all.name; and &a.svn-doc-all.name; are the mailing lists that the version control system uses to send commit messages to. You should never send email directly to these lists. You should only send replies to this list when they are short and are directly related to a commit. &a.developers; All committers are subscribed to -developers. This list was created to be a forum for the committers community issues. Examples are Core voting, announcements, etc. The &a.developers; is for the exclusive use of &os; committers. In order to develop &os;, committers must have the ability to openly discuss matters that will be resolved before they are publicly announced. Frank discussions of work in progress are not suitable for open publication and may harm &os;. All &os; committers are expected not to not publish or forward messages from the &a.developers; outside the list membership without permission of all of the authors. Violators will be removed from the &a.developers;, resulting in a suspension of commit privileges. Repeated or flagrant violations may result in permanent revocation of commit privileges. This list is not intended as a place for code reviews or for any technical discussion. In fact using it as such hurts the &os; Project as it gives a sense of a closed list where general decisions affecting all of the &os; using community are made without being open. Last, but not least never, never ever, email the &a.developers; and CC:/BCC: another &os; list. Never, ever email another &os; email list and CC:/BCC: the &a.developers;. Doing so can greatly diminish the benefits of this list. SSH Quick-Start Guide If you do not wish to type your password in every time you use &man.ssh.1;, and you use RSA or DSA keys to authenticate, &man.ssh-agent.1; is there for your convenience. If you want to use &man.ssh-agent.1;, make sure that you run it before running other applications. X users, for example, usually do this from their .xsession or .xinitrc. See &man.ssh-agent.1; for details. Generate a key pair using &man.ssh-keygen.1;. The key pair will wind up in your $HOME/.ssh/ directory. Send your public key ($HOME/.ssh/id_dsa.pub or $HOME/.ssh/id_rsa.pub) to the person setting you up as a committer so it can be put into yourlogin in /etc/ssh-keys/ on freefall. Now you should be able to use &man.ssh-add.1; for authentication once per session. This will prompt you for your private key's pass phrase, and then store it in your authentication agent (&man.ssh-agent.1;). If you no longer wish to have your key stored in the agent, issuing ssh-add -d will remove it. Test by doing something such as ssh freefall.FreeBSD.org ls /usr. For more information, see security/openssh, &man.ssh.1;, &man.ssh-add.1;, &man.ssh-agent.1;, &man.ssh-keygen.1;, and &man.scp.1;. For information on adding, changing, or removing &man.ssh.1; keys, see this article. &coverity; Availability for &os; Committers All &os; developers can obtain access to Coverity analysis results of all &os; Project software. All who are interested in obtaining access to the analysis results of the automated Coverity runs, can sign up at Coverity Scan. The &os; wiki includes a mini-guide for developers who are interested in working with the &coverity; analysis reports: http://wiki.freebsd.org/CoverityPrevent. Please note that this mini-guide is only readable by &os; developers, so if you cannot access this page, you will have to ask someone to add you to the appropriate Wiki access list. Finally, all &os; developers who are going to use &coverity; are always encouraged to ask for more details and usage information, by posting any questions to the mailing list of the &os; developers. The &os; Committers' Big List of Rules Everyone involved with the &os; project is expected to abide by the Code of Conduct available from http://www.FreeBSD.org/internal/code-of-conduct.html. As committers, you form the public face of the project, and how you behave has a vital impact on the public perception of it. This guide expands on the parts of the Code of Conduct specific to committers. Respect other committers. Respect other contributors. Discuss any significant change before committing. Respect existing maintainers (if listed in the MAINTAINER field in Makefile or in MAINTAINER in the top-level directory). Any disputed change must be backed out pending resolution of the dispute if requested by a maintainer. Security related changes may override a maintainer's wishes at the Security Officer's discretion. Changes go to &os.current; before &os.stable; unless specifically permitted by the release engineer or unless they are not applicable to &os.current;. Any non-trivial or non-urgent change which is applicable should also be allowed to sit in &os.current; for at least 3 days before merging so that it can be given sufficient testing. The release engineer has the same authority over the &os.stable; branch as outlined for the maintainer in rule #5. Do not fight in public with other committers; it looks bad. Respect all code freezes and read the committers and developers mailing lists in a timely manner so you know when a code freeze is in effect. When in doubt on any procedure, ask first! Test your changes before committing them. Do not commit to anything under the src/contrib, src/crypto, or src/sys/contrib trees without explicit approval from the respective maintainer(s). As noted, breaking some of these rules can be grounds for suspension or, upon repeated offense, permanent removal of commit privileges. Individual members of core have the power to temporarily suspend commit privileges until core as a whole has the chance to review the issue. In case of an emergency (a committer doing damage to the repository), a temporary suspension may also be done by the repository meisters. Only a 2/3 majority of core has the authority to suspend commit privileges for longer than a week or to remove them permanently. This rule does not exist to set core up as a bunch of cruel dictators who can dispose of committers as casually as empty soda cans, but to give the project a kind of safety fuse. If someone is out of control, it is important to be able to deal with this immediately rather than be paralyzed by debate. In all cases, a committer whose privileges are suspended or revoked is entitled to a hearing by core, the total duration of the suspension being determined at that time. A committer whose privileges are suspended may also request a review of the decision after 30 days and every 30 days thereafter (unless the total suspension period is less than 30 days). A committer whose privileges have been revoked entirely may request a review after a period of 6 months has elapsed. This review policy is strictly informal and, in all cases, core reserves the right to either act on or disregard requests for review if they feel their original decision to be the right one. In all other aspects of project operation, core is a subset of committers and is bound by the same rules. Just because someone is in core this does not mean that they have special dispensation to step outside any of the lines painted here; core's special powers only kick in when it acts as a group, not on an individual basis. As individuals, the core team members are all committers first and core second. Details Respect other committers. This means that you need to treat other committers as the peer-group developers that they are. Despite our occasional attempts to prove the contrary, one does not get to be a committer by being stupid and nothing rankles more than being treated that way by one of your peers. Whether we always feel respect for one another or not (and everyone has off days), we still have to treat other committers with respect at all times, on public forums and in private email. Being able to work together long term is this project's greatest asset, one far more important than any set of changes to the code, and turning arguments about code into issues that affect our long-term ability to work harmoniously together is just not worth the trade-off by any conceivable stretch of the imagination. To comply with this rule, do not send email when you are angry or otherwise behave in a manner which is likely to strike others as needlessly confrontational. First calm down, then think about how to communicate in the most effective fashion for convincing the other person(s) that your side of the argument is correct, do not just blow off some steam so you can feel better in the short term at the cost of a long-term flame war. Not only is this very bad energy economics, but repeated displays of public aggression which impair our ability to work well together will be dealt with severely by the project leadership and may result in suspension or termination of your commit privileges. The project leadership will take into account both public and private communications brought before it. It will not seek the disclosure of private communications, but it will take it into account if it is volunteered by the committers involved in the complaint. All of this is never an option which the project's leadership enjoys in the slightest, but unity comes first. No amount of code or good advice is worth trading that away. Respect other contributors. You were not always a committer. At one time you were a contributor. Remember that at all times. Remember what it was like trying to get help and attention. Do not forget that your work as a contributor was very important to you. Remember what it was like. Do not discourage, belittle, or demean contributors. Treat them with respect. They are our committers in waiting. They are every bit as important to the project as committers. Their contributions are as valid and as important as your own. After all, you made many contributions before you became a committer. Always remember that. Consider the points raised under and apply them also to contributors. Discuss any significant change before committing. The repository is not where changes should be initially submitted for correctness or argued over, that should happen first in the mailing lists or by use of the Phabricator service and the commit should only happen once something resembling consensus has been reached. This does not mean that you have to ask permission before correcting every obvious syntax error or manual page misspelling, simply that you should try to develop a feel for when a proposed change is not quite such a no-brainer and requires some feedback first. People really do not mind sweeping changes if the result is something clearly better than what they had before, they just do not like being surprised by those changes. The very best way of making sure that you are on the right track is to have your code reviewed by one or more other committers. When in doubt, ask for review! Respect existing maintainers if listed. Many parts of &os; are not owned in the sense that any specific individual will jump up and yell if you commit a change to their area, but it still pays to check first. One convention we use is to put a maintainer line in the Makefile for any package or subtree which is being actively maintained by one or more people; see http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/developers-handbook/policies.html for documentation on this. Where sections of code have several maintainers, commits to affected areas by one maintainer need to be reviewed by at least one other maintainer. In cases where the maintainer-ship of something is not clear, you can also look at the repository logs for the file(s) in question and see if someone has been working recently or predominantly in that area. Other areas of &os; fall under the control of someone who manages an overall category of &os; evolution, such as internationalization or networking. See http://www.FreeBSD.org/administration.html for more information on this. Any disputed change must be backed out pending resolution of the dispute if requested by a maintainer. Security related changes may override a maintainer's wishes at the Security Officer's discretion. This may be hard to swallow in times of conflict (when each side is convinced that they are in the right, of course) but a version control system makes it unnecessary to have an ongoing dispute raging when it is far easier to simply reverse the disputed change, get everyone calmed down again and then try to figure out what is the best way to proceed. If the change turns out to be the best thing after all, it can be easily brought back. If it turns out not to be, then the users did not have to live with the bogus change in the tree while everyone was busily debating its merits. People very rarely call for back-outs in the repository since discussion generally exposes bad or controversial changes before the commit even happens, but on such rare occasions the back-out should be done without argument so that we can get immediately on to the topic of figuring out whether it was bogus or not. Changes go to &os.current; before &os.stable; unless specifically permitted by the release engineer or unless they are not applicable to &os.current;. Any non-trivial or non-urgent change which is applicable should also be allowed to sit in &os.current; for at least 3 days before merging so that it can be given sufficient testing. The release engineer has the same authority over the &os.stable; branch as outlined in rule #5. This is another do not argue about it issue since it is the release engineer who is ultimately responsible (and gets beaten up) if a change turns out to be bad. Please respect this and give the release engineer your full cooperation when it comes to the &os.stable; branch. The management of &os.stable; may frequently seem to be overly conservative to the casual observer, but also bear in mind the fact that conservatism is supposed to be the hallmark of &os.stable; and different rules apply there than in &os.current;. There is also really no point in having &os.current; be a testing ground if changes are merged over to &os.stable; immediately. Changes need a chance to be tested by the &os.current; developers, so allow some time to elapse before merging unless the &os.stable; fix is critical, time sensitive or so obvious as to make further testing unnecessary (spelling fixes to manual pages, obvious bug/typo fixes, etc.) In other words, apply common sense. Changes to the security branches (for example, releng/9.3) must be approved by a member of the &a.security-officer;, or in some cases, by a member of the &a.re;. Do not fight in public with other committers; it looks bad. This project has a public image to uphold and that image is very important to all of us, especially if we are to continue to attract new members. There will be occasions when, despite everyone's very best attempts at self-control, tempers are lost and angry words are exchanged. The best thing that can be done in such cases is to minimize the effects of this until everyone has cooled back down. That means that you should not air your angry words in public and you should not forward private correspondence or other private communications to public mailing lists, mail aliases, instant messaging channels or social media sites. What people say one-to-one is often much less sugar-coated than what they would say in public, and such communications therefore have no place there - they only serve to inflame an already bad situation. If the person sending you a flame-o-gram at least had the grace to send it privately, then have the grace to keep it private yourself. If you feel you are being unfairly treated by another developer, and it is causing you anguish, bring the matter up with core rather than taking it public. Core will do its best to play peace makers and get things back to sanity. In cases where the dispute involves a change to the codebase and the participants do not appear to be reaching an amicable agreement, core may appoint a mutually-agreeable third party to resolve the dispute. All parties involved must then agree to be bound by the decision reached by this third party. Respect all code freezes and read the committers and developers mailing list on a timely basis so you know when a code freeze is in effect. Committing unapproved changes during a code freeze is a really big mistake and committers are expected to keep up-to-date on what is going on before jumping in after a long absence and committing 10 megabytes worth of accumulated stuff. People who abuse this on a regular basis will have their commit privileges suspended until they get back from the &os; Happy Reeducation Camp we run in Greenland. When in doubt on any procedure, ask first! Many mistakes are made because someone is in a hurry and just assumes they know the right way of doing something. If you have not done it before, chances are good that you do not actually know the way we do things and really need to ask first or you are going to completely embarrass yourself in public. There is no shame in asking how in the heck do I do this? We already know you are an intelligent person; otherwise, you would not be a committer. Test your changes before committing them. This may sound obvious, but if it really were so obvious then we probably would not see so many cases of people clearly not doing this. If your changes are to the kernel, make sure you can still compile both GENERIC and LINT. If your changes are anywhere else, make sure you can still make world. If your changes are to a branch, make sure your testing occurs with a machine which is running that code. If you have a change which also may break another architecture, be sure and test on all supported architectures. Please refer to the &os; Internal Page for a list of available resources. As other architectures are added to the &os; supported platforms list, the appropriate shared testing resources will be made available. Do not commit to anything under the src/contrib, src/crypto, and src/sys/contrib trees without explicit approval from the respective maintainer(s). The trees mentioned above are for contributed software usually imported onto a vendor branch. Committing something there, even if it does not take the file off the vendor branch, may cause unnecessary headaches for those responsible for maintaining that particular piece of software. Thus, unless you have explicit approval from the maintainer (or you are the maintainer), do not commit there! Please note that this does not mean you should not try to improve the software in question; you are still more than welcome to do so. Ideally, you should submit your patches to the vendor. If your changes are &os;-specific, talk to the maintainer; they may be willing to apply them locally. But whatever you do, do not commit there by yourself! Contact the &a.core; if you wish to take up maintainership of an unmaintained part of the tree. Policy on Multiple Architectures &os; has added several new architecture ports during recent release cycles and is truly no longer an &i386; centric operating system. In an effort to make it easier to keep &os; portable across the platforms we support, core has developed the following mandate:
Our 32-bit reference platform is &arch.i386;, and our 64-bit reference platform is &arch.sparc64;. Major design work (including major API and ABI changes) must prove itself on at least one 32-bit and at least one 64-bit platform, preferably the primary reference platforms, before it may be committed to the source tree.
The &arch.i386; and &arch.sparc64; platforms were chosen due to being more readily available to developers and as representatives of more diverse processor and system designs - big versus little endian, register file versus register stack, different DMA and cache implementations, hardware page tables versus software TLB management etc. The &arch.ia64; platform has many of the same complications that &arch.sparc64; has, but is still limited in availability to developers. We will continue to re-evaluate this policy as cost and availability of the 64-bit platforms change. Developers should also be aware of our Tier Policy for the long term support of hardware architectures. The rules here are intended to provide guidance during the development process, and are distinct from the requirements for features and architectures listed in that section. The Tier rules for feature support on architectures at release-time are more strict than the rules for changes during the development process. Other Suggestions When committing documentation changes, use a spell checker before committing. For all XML docs, verify that the formatting directives are correct by running make lint and textproc/igor. For manual pages, run sysutils/manck and textproc/igor over the manual page to verify all of the cross references and file references are correct and that the man page has all of the appropriate MLINKs installed. Do not mix style fixes with new functionality. A style fix is any change which does not modify the functionality of the code. Mixing the changes obfuscates the functionality change when asking for differences between revisions, which can hide any new bugs. Do not include whitespace changes with content changes in commits to doc/ . The extra clutter in the diffs makes the translators' job much more difficult. Instead, make any style or whitespace changes in separate commits that are clearly labeled as such in the commit message. Deprecating Features When it is necessary to remove functionality from software in the base system the following guidelines should be followed whenever possible: Mention is made in the manual page and possibly the release notes that the option, utility, or interface is deprecated. Use of the deprecated feature generates a warning. The option, utility, or interface is preserved until the next major (point zero) release. The option, utility, or interface is removed and no longer documented. It is now obsolete. It is also generally a good idea to note its removal in the release notes. Privacy and Confidentiality Most &os; business is done in public. &os; is an open project. Which means that not only can anyone use the source code, but that most of the development process is open to public scrutiny. Certain sensitive matters must remain private or held under embargo. There unfortunately cannot be complete transparency. As a &os; developer you will have a certain degree of privileged access to information. Consequently you are expected to respect certain requirements for confidentiality. Sometimes the need for confidentiality comes from external collaborators or has a specific time limit. Mostly though, it is a matter of not releasing private communications. The Security Officer has sole control over the release of security advisories. Where there are security problems that affect many different operating systems, &os; frequently depends on early access in order to be able to prepare advisories for coordinated release. Unless &os; developers can be trusted to maintain security, such early access will not be made available. The Security Officer is responsible for controlling pre-release access to information about vulnerabilities, and for timing the release of all advisories. He may request help under condition of confidentiality from any developer with relevant knowledge in order to prepare security fixes. Communications with Core are kept confidential for as long as necessary. Communications to core will initially be treated as confidential. Eventually however, most of Core's business will be summarized into the monthly or quarterly core reports. Care will be taken to avoid publicising any sensitive details. Records of some particularly sensitive subjects may not be reported on at all and will be retained only in Core's private archives. Non-disclosure Agreements may be required for access to certain commercially sensitive data. Access to certain commercially sensitive data may only be available under a Non-Disclosure Agreement. The FreeBSD Foundation legal staff must be consulted before any binding agreements are entered into. Private communications should not be made public without permission. Beyond the specific requirements above there is a general expectation not to publish private communications between developers without the consent of all parties involved. Ask permission before forwarding a message onto a public mailing list, or posting it to a forum or website that can be accessed by other than the original correspondents. Communications on project-only or restricted access channels should be treated as private. Similarly to personal communications, certain internal communications channels, including &os; Committer only mailing lists and restricted access IRC channels should be considered as private communications. You need permission in order to publish material from these sources. Core may approve publication. Where it is impractical to obtain permission due to the number of correspondents or where permission to publish is unreasonably withheld, Core may approve release of such private matters that merit more general publication. Support for Multiple Architectures &os; is a highly portable operating system intended to function on many different types of hardware architectures. Maintaining clean separation of Machine Dependent (MD) and Machine Independent (MI) code, as well as minimizing MD code, is an important part of our strategy to remain agile with regards to current hardware trends. Each new hardware architecture supported by &os; adds substantially to the cost of code maintenance, toolchain support, and release engineering. It also dramatically increases the cost of effective testing of kernel changes. As such, there is strong motivation to differentiate between classes of support for various architectures while remaining strong in a few key architectures that are seen as the &os; target audience. Statement of General Intent The &os; Project targets "production quality commercial off-the-shelf (COTS) workstation, server, and high-end embedded systems". By retaining a focus on a narrow set of architectures of interest in these environments, the &os; Project is able to maintain high levels of quality, stability, and performance, as well as minimize the load on various support teams on the project, such as the ports team, documentation team, security officer, and release engineering teams. Diversity in hardware support broadens the options for &os; consumers by offering new features and usage opportunities (such as support for 64-bit CPUs, use in embedded environments, etc.), but these benefits must always be carefully considered in terms of the real-world maintenance cost associated with additional platform support. The &os; Project differentiates platform targets into four tiers. Each tier includes a specification of the requirements for an architecture to be in that tier, as well as specifying the obligations of developers with regards to the platform. In addition, a policy is defined regarding the circumstances required to change the tier of an architecture. Tier 1: Fully Supported Architectures Tier 1 platforms are fully supported by the security officer, release engineering, and toolchain maintenance staff. New features added to the operating system must be fully functional across all Tier 1 architectures for every release (features which are inherently architecture-specific, such as support for hardware device drivers, may be exempt from this requirement). In general, all Tier 1 platforms must have build and Tinderbox support either in the FreeBSD.org cluster, or be easily available for all developers. Embedded platforms may substitute an emulator available in the &os; cluster for actual hardware. Tier 1 architectures are expected to be Production Quality with respects to all aspects of the &os; operating system, including installation and development environments. Tier 1 architectures are expected to be completely integrated into the source tree and have all features necessary to produce an entire system relevant for that target architecture. Tier 1 architectures generally have at least 6 active developers. Tier 1 architectures are expected to be fully supported by the ports system. All the ports should build on a Tier 1 platform, or have the appropriate filters to prevent the inappropriate ones from building there. The packaging system must support all Tier 1 architectures. To ensure an architecture's Tier 1 status, proponents of that architecture must show that all relevant packages can be built on that platform. Tier 1 embedded architectures must be able to cross-build packages on at least one other Tier 1 architecture. The packages must be the most relevant for the platform, but may be a non-empty subset of those that build natively. Tier 1 architectures must be fully documented. All basic operations need to be covered by the handbook or other documents. All relevant integration documentation must also be integrated into the tree, or readily available. Current Tier 1 platforms are &arch.i386; and &arch.amd64;. Tier 2: Developmental Architectures Tier 2 platforms are not supported by the security officer and release engineering teams. Platform maintainers are responsible for toolchain support in the tree. The toolchain maintainers are expected to work with the platform maintainers to refine these changes. Major new toolchain components are allowed to break support for Tier 2 architectures if the &os;-local changes have not been incorporated upstream. The toolchain maintainers are expected to provide prompt review of any proposed changes and cannot block, through their inaction, changes going into the tree. New features added to &os; should be feasible to implement on these platforms, but an implementation is not required before the feature may be added to the &os; source tree. New features that may be difficult to implement on Tier 2 architectures should provide a means of disabling them on those architectures. The implementation of a Tier 2 architecture may be committed to the main &os; tree as long as it does not interfere with production work on Tier 1 platforms, or substantially with other Tier 2 platforms. Before a Tier 2 platform can be added to the &os; base source tree, the platform must be able to boot multi-user on actual hardware. Generally, there must be at least three active developers working on the platform. Tier 2 architectures are usually systems targeted at Tier 1 support, but that are still under development. Architectures reaching end of life may also be moved from Tier 1 status to Tier 2 status as the availability of resources to continue to maintain the system in a Production Quality state diminishes. Well supported niche architectures may also be Tier 2. Tier 2 architectures have basic support for them integrated into the ports infrastructure. They may have cross compilation support added, at the discretion of portmgr. Some ports must built natively into packages if the package system supports that architecture. If not integrated into the base system, some external patches for the architecture for ports must be available. Tier 2 architectures can be integrated into the &os; handbook. The basics for how to get a system running must be documented, although not necessarily for every single board or system a Tier 2 architecture supports. The supported hardware list must exist and should be relatively recent. It should be integrated into the &os; documentation. Current Tier 2 platforms are &arch.arm;, &arch.ia64; (through &os; 10), &arch.pc98;, &arch.powerpc;, and &arch.sparc64;. Tier 3: Experimental Architectures Tier 3 platforms are not supported by the security officer and release engineering teams. At the discretion of the toolchain maintainers, they may be supported in the toolchain. Tier 3 platforms are architectures in the early stages of development, for non-mainstream hardware platforms, or which are considered legacy systems unlikely to see broad future use. Initial support for Tier 3 platforms should be worked on in external SCM repositories. The transition to &os;'s subversion should take place after the platform boots multi-user on hardware; sharing via subversion is needed for wider exposure; and multiple developers are actively working on the platform. Platforms that transition to Tier 3 status may be removed from the tree if they are no longer actively supported by the &os; developer community at the discretion of the release engineer. Tier 3 platforms may have ports support, either integrated or external, but do not require it. Tier 3 platforms must have the basics documented for how to build a kernel and how to boot it on at least one target hardware or emulation environment. This documentation need not be integrated into the &os; tree. Current Tier 3 platforms are &arch.mips;. Tier 4: Unsupported Architectures Tier 4 systems are not supported in any form by the project. All systems not otherwise classified into a support tier are Tier 4 systems. The &arch.ia64; platform is transitioning to Tier 4 status in &os; 11. Policy on Changing the Tier of an Architecture Systems may only be moved from one tier to another by approval of the &os; Core Team, which shall make that decision in collaboration with the Security Officer, Release Engineering, and toolchain maintenance teams. Ports Specific FAQ Adding a New Port How do I add a new port? First, please read the section about repository copies. The easiest way to add a new port is the addport script located in the ports/Tools/scripts directory. It adds a port from the directory specified, determining the category automatically from the port Makefile. It also adds an entry to the port's category Makefile. It was written by &a.mharo.email;, &a.will.email;, and &a.garga.email;. When sending questions about this script to the &a.ports;, please also CC &a.crees.email;, the current maintainer. Any other things I need to know when I add a new port? Check the port, preferably to make sure it compiles and packages correctly. This is the recommended sequence: &prompt.root; make install &prompt.root; make package &prompt.root; make deinstall &prompt.root; pkg add package you built above &prompt.root; make deinstall &prompt.root; make reinstall &prompt.root; make package The Porters Handbook contains more detailed instructions. Use &man.portlint.1; to check the syntax of the port. You do not necessarily have to eliminate all warnings but make sure you have fixed the simple ones. If the port came from a submitter who has not contributed to the Project before, add that person's name to the Additional Contributors section of the &os; Contributors List. Close the PR if the port came in as a PR. To close a PR, change the state to Issue Resolved and the resolution as Fixed. Removing an Existing Port How do I remove an existing port? First, please read the section about repository copies. Before you remove the port, you have to verify there are no other ports depending on it. Make sure there is no dependency on the port in the ports collection: The port's PKGNAME should appear in exactly one line in a recent INDEX file. No other ports should contain any reference to the port's directory or PKGNAME in their Makefiles Then, remove the port: Remove the port's files and directory with svn remove. Remove the SUBDIR listing of the port in the parent directory Makefile. Add an entry to ports/MOVED. Remove the port from ports/LEGAL if it is there. Alternatively, you can use the rmport script, from ports/Tools/scripts. This script was written by &a.vd.email;. When sending questions about this script to the &a.ports;, please also CC &a.crees.email;, the current maintainer. Re-adding a Deleted Port How do I re-add a deleted port? This is essentially the reverse of deleting a port. Do not use svn add to add the port. Follow these steps. If they are unclear, or are not working, ask for help, do not just svn add the port. Figure out when the port was removed. Use this list, or look for the port on freshports, and then copy the last living revision of the port: &prompt.user; cd /usr/ports/category &prompt.user; svn cp 'svn+ssh://repo.freebsd.org/ports/head/category/portname/@XXXXXX' portname Pick the revision that is just before the removal. For example, if the revision where it was removed is 269874, use 269873. It is also possible to specify a date. In that case, pick a date that is before the removal but after the last commit to the port. &prompt.user; cd /usr/ports/category &prompt.user; svn cp 'svn+ssh://repo.freebsd.org/ports/head/category/portname/@{YYYY-MM-DD}' portname Make the changes necessary to get the port working again. If it was deleted because the distfiles are no longer available, either volunteer to host the distfiles, or find someone else to do so. If some files have been added, or were removed during the resurrection process, use svn add or svn remove to make sure all the files in the port will be committed. Restore the SUBDIR listing of the port in the parent directory Makefile, keeping the entries sorted. Delete the port entry from ports/MOVED. If the port had an entry in ports/LEGAL, restore it. svn commit these changes, preferably in one step. The addport script mentioned in now detects when the port to add has previously existed, and attempts to handle all except the ports/LEGAL step automatically. Repository Copies When do we need a repository copy? When you want to add a port that is related to any port that is already in the tree in a separate directory, you have to do a repository copy. Here related means it is a different version or a slightly modified version. Examples are print/ghostscript* (different versions) and x11-wm/windowmaker* (English-only and internationalized version). Another example is when a port is moved from one subdirectory to another, or when you want to change the name of a directory because the author(s) renamed their software even though it is a descendant of a port already in a tree. What do I need to do? With Subversion, a repo copy can be done by any committer: Doing a repo copy: Verify that the target directory does not exist. Use svn up to make certain the original files, directories, and checkout information is current. Use svn move or svn copy to do the repo copy. Upgrade the copied port to the new version. Remember to add or change the PKGNAMEPREFIX or PKGNAMESUFFIX so there are no duplicate ports with the same name. In some rare cases it may be necessary to change the PORTNAME instead of adding PKGNAMEPREFIX or PKGNAMESUFFIX, but this should only be done when it is really needed — e.g., using an existing port as the base for a very similar program with a different name, or upgrading a port to a new upstream version which actually changes the distribution name, like the transition from textproc/libxml to textproc/libxml2. In most cases, adding or changing PKGNAMEPREFIX or PKGNAMESUFFIX should suffice. Add the new subdirectory to the SUBDIR listing in the parent directory Makefile. You can run make checksubdirs in the parent directory to check this. If the port changed categories, modify the CATEGORIES line of the port's Makefile accordingly Add an entry to ports/MOVED, if you remove the original port. Commit all changes on one commit. When removing a port: Perform a thorough check of the ports collection for any dependencies on the old port location/name, and update them. Running grep on INDEX is not enough because some ports have dependencies enabled by compile-time options. A full grep -r of the ports collection is recommended. Remove the old port and the old SUBDIR entry. Add an entry to ports/MOVED. After repo moves (rename operations where a port is copied and the old location is removed): Follow the same steps that are outlined in the previous two entries, to activate the new location of the port and remove the old one. Ports Freeze What is a ports freeze? A ports freeze was a restricted state the ports tree was put in before a release. It was used to ensure a higher quality for the packages shipped with a release. It usually lasted a couple of weeks. During that time, build problems were fixed, and the release packages were built. This practice is no longer used, as the packages for the releases are built from the current stable, quarterly branch. For more information on how to merge commits to the quarterly branch, see . Creating a New Category What is the procedure for creating a new category? Please see Proposing a New Category in the Porter's Handbook. Once that procedure has been followed and the PR has been assigned to &a.portmgr;, it is their decision whether or not to approve it. If they do, it is their responsibility to do the following: Perform any needed moves. (This only applies to physical categories.) Update the VALID_CATEGORIES definition in ports/Mk/bsd.port.mk. Assign the PR back to you. What do I need to do to implement a new physical category? Upgrade each moved port's Makefile. Do not connect the new category to the build yet. To do this, you will need to: Change the port's CATEGORIES (this was the point of the exercise, remember?) The new category should be listed first. This will help to ensure that the PKGORIGIN is correct. Run a make describe. Since the top-level make index that you will be running in a few steps is an iteration of make describe over the entire ports hierarchy, catching any errors here will save you having to re-run that step later on. If you want to be really thorough, now might be a good time to run &man.portlint.1;. Check that the PKGORIGINs are correct. The ports system uses each port's CATEGORIES entry to create its PKGORIGIN, which is used to connect installed packages to the port directory they were built from. If this entry is wrong, common port tools like &man.pkg.version.1; and &man.portupgrade.1; fail. To do this, use the chkorigin.sh tool, as follows: env PORTSDIR=/path/to/ports sh -e /path/to/ports/Tools/scripts/chkorigin.sh. This will check every port in the ports tree, even those not connected to the build, so you can run it directly after the move operation. Hint: do not forget to look at the PKGORIGINs of any slave ports of the ports you just moved! On your own local system, test the proposed changes: first, comment out the SUBDIR entries in the old ports' categories' Makefiles; then enable building the new category in ports/Makefile. Run make checksubdirs in the affected category directories to check the SUBDIR entries. Next, in the ports/ directory, run make index. This can take over 40 minutes on even modern systems; however, it is a necessary step to prevent problems for other people. Once this is done, you can commit the updated ports/Makefile to connect the new category to the build and also commit the Makefile changes for the old category or categories. Add appropriate entries to ports/MOVED. Update the documentation by modifying the following: the list of categories in the Porter's Handbook doc/en_US.ISO8859-1/htdocs/ports. Note that these are now displayed by sub-groups, as specified in doc/en_US.ISO8859-1/htdocs/ports/categories.descriptions. (Note: these are in the docs, not the ports, repository). If you are not a docs committer, you will need to submit a PR for this. Only once all the above have been done, and no one is any longer reporting problems with the new ports, should the old ports be deleted from their previous locations in the repository. It is not necessary to manually update the ports web pages to reflect the new category. This is done automatically via the change to en_US.ISO8859-1/htdocs/ports/categories and the automated rebuild of INDEX. What do I need to do to implement a new virtual category? This is much simpler than a physical category. You only need to modify the following: the list of categories in the Porter's Handbook en_US.ISO8859-1/htdocs/ports/categories Miscellaneous Questions How do I know if my port is building correctly or not? The packages are built multiple times each week. If a port fails, the maintainer will receive an email from pkg-fallout@FreeBSD.org. Reports for all the package builds (official, experimental, and non-regression) are aggregated at pkg-status.FreeBSD.org. I added a new port. Do I need to add it to the INDEX? No. The file can either be generated by running make index, or a pre-generated version can be downloaded with make fetchindex. Are there any other files I am not allowed to touch? Any file directly under ports/, or any file under a subdirectory that starts with an uppercase letter (Mk/, Tools/, etc.). In particular, the &a.portmgr; is very protective of ports/Mk/bsd.port*.mk so do not commit changes to those files unless you want to face their wra(i)th. What is the proper procedure for updating the checksum for a port's distfile when the file changes without a version change? When the checksum for a port's distfile is updated due to the author updating the file without changing the port's revision, the commit message should include a summary of the relevant diffs between the original and new distfile to ensure that the distfile has not been corrupted or maliciously altered. If the current version of the port has been in the ports tree for a while, a copy of the old distfile will usually be available on the ftp servers; otherwise the author or maintainer should be contacted to find out why the distfile has changed. What is the procedure to request authorization for merging a commit to the quarterly branch? When doing the commit, add the branch name to the MFH: line, for example: MFH: 2014Q1 It will automatically notify &a.ports-secteam; and &a.portmgr;. They will then decide if the commit can be merged and answer with the procedure. If the commit has already been made, send an email to &a.ports-secteam; and &a.portmgr; with the revision number and a small description of why the commit needs to be merged. A script is provided to automate merging a specific commit: ports/Tools/scripts/mfh. It is used as follows: &prompt.user; /usr/ports/Tools/scripts/mfh 2015Q1 380362 U 2015Q1 Checked out revision 380443. A 2015Q1/security Updating '2015Q1/security/rubygem-sshkit': A 2015Q1/security/rubygem-sshkit A 2015Q1/security/rubygem-sshkit/Makefile A 2015Q1/security/rubygem-sshkit/distinfo A 2015Q1/security/rubygem-sshkit/pkg-descr Updated to revision 380443. --- Merging r380362 into '2015Q1': U 2015Q1/security/rubygem-sshkit/Makefile U 2015Q1/security/rubygem-sshkit/distinfo --- Recording mergeinfo for merge of r380362 into '2015Q1': U 2015Q1 --- Recording mergeinfo for merge of r380362 into '2015Q1/security': G 2015Q1/security --- Eliding mergeinfo from '2015Q1/security': U 2015Q1/security --- Recording mergeinfo for merge of r380362 into '2015Q1/security/rubygem-sshkit': G 2015Q1/security/rubygem-sshkit --- Eliding mergeinfo from '2015Q1/security/rubygem-sshkit': U 2015Q1/security/rubygem-sshkit M 2015Q1 M 2015Q1/security/rubygem-sshkit/Makefile M 2015Q1/security/rubygem-sshkit/distinfo Index: 2015Q1/security/rubygem-sshkit/Makefile =================================================================== --- 2015Q1/security/rubygem-sshkit/Makefile (revision 380443) +++ 2015Q1/security/rubygem-sshkit/Makefile (working copy) @@ -2,7 +2,7 @@ # $FreeBSD$ PORTNAME= sshkit -PORTVERSION= 1.6.1 +PORTVERSION= 1.7.0 CATEGORIES= security rubygems MASTER_SITES= RG Index: 2015Q1/security/rubygem-sshkit/distinfo =================================================================== --- 2015Q1/security/rubygem-sshkit/distinfo (revision 380443) +++ 2015Q1/security/rubygem-sshkit/distinfo (working copy) @@ -1,2 +1,2 @@ -SHA256 (rubygem/sshkit-1.6.1.gem) = 8ca67e46bb4ea50fdb0553cda77552f3e41b17a5aa919877d93875dfa22c03a7 -SIZE (rubygem/sshkit-1.6.1.gem) = 135680 +SHA256 (rubygem/sshkit-1.7.0.gem) = 90effd1813363bae7355f4a45ebc8335a8ca74acc8d0933ba6ee6d40f281a2cf +SIZE (rubygem/sshkit-1.7.0.gem) = 136192 Index: 2015Q1 =================================================================== --- 2015Q1 (revision 380443) +++ 2015Q1 (working copy) Property changes on: 2015Q1 ___________________________________________________________________ Modified: svn:mergeinfo Merged /head:r380362 Do you want to commit? (no = start a shell) [y/n] At that point, the script will either open a shell for you to fix things, or open your text editor with the commit message all prepared and then commit the merge. The script assumes that you can connect to repo.FreeBSD.org with SSH directly, so if your local login name is different than your &os; cluster account, you need a few lines in your ~/.ssh/config: Host repo.freebsd.org # Can be *.freebsd.org User freebsd-login Issues Specific to Developers Who Are Not Committers A few people who have access to the &os; machines do not have commit bits. Almost all of this document will apply to these developers as well (except things specific to commits and the mailing list memberships that go with them). In particular, we recommend that you read: Administrative Details Conventions You should get your mentor to add you to the Additional Contributors (doc/en_US.ISO8859-1/articles/contributors/contrib.additional.xml), if you are not already listed there. Developer Relations SSH Quick-Start Guide The &os; Committers' Big List of Rules Information About &ga; As of December 12, 2012, &ga; was enabled on the &os; Project website to collect anonymized usage statistics regarding usage of the site. The information collected is valuable to the &os; Documentation Project, in order to identify various problems on the &os; website. &ga; General Policy The &os; Project takes visitor privacy very seriously. As such, the &os; Project website honors the Do Not Track header before fetching the tracking code from Google. For more information, please see the &os; Privacy Policy. &ga; access is not arbitrarily allowed — access must be requested, voted on by the &a.doceng;, and explicitly granted. Requests for &ga; data must include a specific purpose. For example, a valid reason for requesting access would be to see the most frequently used web browsers when viewing &os; web pages to ensure page rendering speeds are acceptable. Conversely, to see what web browsers are most frequently used (without stating why) would be rejected. All requests must include the timeframe for which the data would be required. For example, it must be explicitly stated if the requested data would be needed for a timeframe covering a span of 3 weeks, or if the request would be one-time only. Any request for &ga; data without a clear, reasonable reason beneficial to the &os; Project will be rejected. Data Available Through &ga; A few examples of the types of &ga; data available include: Commonly used web browsers Page load times Site access by language Miscellaneous Questions Why are trivial or cosmetic changes to files on a vendor branch a bad idea? From now on, every new vendor release of that file will need to have patches merged in by hand. From now on, every new vendor release of that file will need to have patches verified by hand. How do I add a new file to a branch? To add a file onto a branch, simply checkout or update to the branch you want to add to and then add the file using the add operation as you normally would. This works fine for the doc and ports trees. The src tree uses SVN and requires more care because of the mergeinfo properties. See the Subversion Primer for details on how to perform an MFC. How do I access people.FreeBSD.org to put up personal or project information? people.FreeBSD.org is the same as freefall.FreeBSD.org. Just create a public_html directory. Anything you place in that directory will automatically be visible under http://people.FreeBSD.org/. Where are the mailing list archives stored? The mailing lists are archived under /local/mail on freefall.FreeBSD.org. I would like to mentor a new committer. What process do I need to follow? See the New Account Creation Procedure document on the internal pages. Are there any perks of being an &os; committer? Recognition as a competent software engineer is the longest lasting value. In addition, getting a chance to work with some of the best people that every engineer would dream of meeting is a great perk! &os; committers can get a free 4-CD or DVD set at conferences from &os; Mall, Inc.. In addition, developers may request a cloaked hostmask for their account on the Freenode IRC network in the form of freebsd/developer/freefall name or freebsd/developer/NickServ name. To request a cloak, send an email to &a.irc.email; with your requested hostmask and NickServ account name.
Index: head/en_US.ISO8859-1/books/handbook/cutting-edge/chapter.xml =================================================================== --- head/en_US.ISO8859-1/books/handbook/cutting-edge/chapter.xml (revision 47723) +++ head/en_US.ISO8859-1/books/handbook/cutting-edge/chapter.xml (revision 47724) @@ -1,2243 +1,2240 @@ Updating and Upgrading &os; Jim Mock Restructured, reorganized, and parts updated by Jordan Hubbard Original work by Poul-Henning Kamp John Polstra Nik Clayton Synopsis &os; is under constant development between releases. Some people prefer to use the officially released versions, while others prefer to keep in sync with the latest developments. However, even official releases are often updated with security and other critical fixes. Regardless of the version used, &os; provides all the necessary tools to keep the system updated, and allows for easy upgrades between versions. This chapter describes how to track the development system and the basic tools for keeping a &os; system up-to-date. After reading this chapter, you will know: How to keep a &os; system up-to-date with freebsd-update, Subversion, or CTM. How to compare the state of an installed system against a known pristine copy. How to keep the installed documentation up-to-date with Subversion or documentation ports. The difference between the two development branches: &os.stable; and &os.current;. How to rebuild and reinstall the entire base system. Before reading this chapter, you should: Properly set up the network connection (). Know how to install additional third-party software (). Throughout this chapter, svn is used to obtain and update &os; sources. To use it, first install the devel/subversion port or package. &os; Update Tom Rhodes Written by Colin Percival Based on notes provided by Updating and Upgrading freebsd-update updating-upgrading Applying security patches in a timely manner and upgrading to a newer release of an operating system are important aspects of ongoing system administration. &os; includes a utility called freebsd-update which can be used to perform both these tasks. This utility supports binary security and errata updates to &os;, without the need to manually compile and install the patch or a new kernel. Binary updates are available for all architectures and releases currently supported by the security team. The list of supported releases and their estimated end-of-life dates are listed at http://www.FreeBSD.org/security/. This utility also supports operating system upgrades to minor point releases as well as upgrades to another release branch. Before upgrading to a new release, review its release announcement as it contains important information pertinent to the release. Release announcements are available from http://www.FreeBSD.org/releases/. If a crontab utilizing the features of &man.freebsd-update.8; exists, it must be disabled before upgrading the operating system. This section describes the configuration file used by freebsd-update, demonstrates how to apply a security patch and how to upgrade to a minor or major operating system release, and discusses some of the considerations when upgrading the operating system. The Configuration File The default configuration file for freebsd-update works as-is. Some users may wish to tweak the default configuration in /etc/freebsd-update.conf, allowing better control of the process. The comments in this file explain the available options, but the following may require a bit more explanation: # Components of the base system which should be kept updated. Components world kernel This parameter controls which parts of &os; will be kept up-to-date. The default is to update the entire base system and the kernel. Individual components can instead be specified, such as src/base or src/sys. However, the best option is to leave this at the default as changing it to include specific items requires every needed item to be listed. Over time, this could have disastrous consequences as source code and binaries may become out of sync. # Paths which start with anything matching an entry in an IgnorePaths # statement will be ignored. IgnorePaths /boot/kernel/linker.hints To leave specified directories, such as /bin or /sbin, untouched during the update process, add their paths to this statement. This option may be used to prevent freebsd-update from overwriting local modifications. # Paths which start with anything matching an entry in an UpdateIfUnmodified # statement will only be updated if the contents of the file have not been # modified by the user (unless changes are merged; see below). UpdateIfUnmodified /etc/ /var/ /root/ /.cshrc /.profile This option will only update unmodified configuration files in the specified directories. Any changes made by the user will prevent the automatic updating of these files. There is another option, KeepModifiedMetadata, which will instruct freebsd-update to save the changes during the merge. # When upgrading to a new &os; release, files which match MergeChanges # will have any local changes merged into the version from the new release. MergeChanges /etc/ /var/named/etc/ /boot/device.hints List of directories with configuration files that freebsd-update should attempt to merge. The file merge process is a series of &man.diff.1; patches similar to &man.mergemaster.8;, but with fewer options. Merges are either accepted, open an editor, or cause freebsd-update to abort. When in doubt, backup /etc and just accept the merges. See for more information about mergemaster. # Directory in which to store downloaded updates and temporary # files used by &os; Update. # WorkDir /var/db/freebsd-update This directory is where all patches and temporary files are placed. In cases where the user is doing a version upgrade, this location should have at least a gigabyte of disk space available. # When upgrading between releases, should the list of Components be # read strictly (StrictComponents yes) or merely as a list of components # which *might* be installed of which &os; Update should figure out # which actually are installed and upgrade those (StrictComponents no)? # StrictComponents no When this option is set to yes, freebsd-update will assume that the Components list is complete and will not attempt to make changes outside of the list. Effectively, freebsd-update will attempt to update every file which belongs to the Components list. Applying Security Patches The process of applying &os; security patches has been simplified, allowing an administrator to keep a system fully patched using freebsd-update. More information about &os; security advisories can be found in . &os; security patches may be downloaded and installed using the following commands. The first command will determine if any outstanding patches are available, and if so, will list the files that will be modifed if the patches are applied. The second command will apply the patches. &prompt.root; freebsd-update fetch &prompt.root; freebsd-update install If the update applies any kernel patches, the system will need a reboot in order to boot into the patched kernel. If the patch was applied to any running binaries, the affected applications should be restarted so that the patched version of the binary is used. The system can be configured to automatically check for updates once every day by adding this entry to /etc/crontab: @daily root freebsd-update cron If patches exist, they will automatically be downloaded but will not be applied. The root user will be sent an email so that the patches may be reviewed and manually installed with freebsd-update install. If anything goes wrong, freebsd-update has the ability to roll back the last set of changes with the following command: &prompt.root; freebsd-update rollback Uninstalling updates... done. Again, the system should be restarted if the kernel or any kernel modules were modified and any affected binaries should be restarted. Only the GENERIC kernel can be automatically updated by freebsd-update. If a custom kernel is installed, it will have to be rebuilt and reinstalled after freebsd-update finishes installing the updates. However, freebsd-update will detect and update the GENERIC kernel if /boot/GENERIC exists, even if it is not the current running kernel of the system. Always keep a copy of the GENERIC kernel in /boot/GENERIC. It will be helpful in diagnosing a variety of problems and in performing version upgrades. Refer to for instructions on how to get a copy of the GENERIC kernel. Unless the default configuration in /etc/freebsd-update.conf has been changed, freebsd-update will install the updated kernel sources along with the rest of the updates. Rebuilding and reinstalling a new custom kernel can then be performed in the usual way. The updates distributed by freebsd-update do not always involve the kernel. It is not necessary to rebuild a custom kernel if the kernel sources have not been modified by freebsd-update install. However, freebsd-update will always update /usr/src/sys/conf/newvers.sh. The current patch level, as indicated by the -p number reported by uname -r, is obtained from this file. Rebuilding a custom kernel, even if nothing else changed, allows uname to accurately report the current patch level of the system. This is particularly helpful when maintaining multiple systems, as it allows for a quick assessment of the updates installed in each one. Performing Major and Minor Version Upgrades Upgrades from one minor version of &os; to another, like from &os; 9.0 to &os; 9.1, are called minor version upgrades. Major version upgrades occur when &os; is upgraded from one major version to another, like from &os; 9.X to &os; 10.X. Both types of upgrades can be performed by providing freebsd-update with a release version target. If the system is running a custom kernel, make sure that a copy of the GENERIC kernel exists in /boot/GENERIC before starting the upgrade. Refer to for instructions on how to get a copy of the GENERIC kernel. The following command, when run on a &os; 9.0 system, will upgrade it to &os; 9.1: &prompt.root; freebsd-update -r 9.1-RELEASE upgrade After the command has been received, freebsd-update will evaluate the configuration file and current system in an attempt to gather the information necessary to perform the upgrade. A screen listing will display which components have and have not been detected. For example: Looking up update.FreeBSD.org mirrors... 1 mirrors found. Fetching metadata signature for 9.0-RELEASE from update1.FreeBSD.org... done. Fetching metadata index... done. Inspecting system... done. The following components of FreeBSD seem to be installed: kernel/smp src/base src/bin src/contrib src/crypto src/etc src/games src/gnu src/include src/krb5 src/lib src/libexec src/release src/rescue src/sbin src/secure src/share src/sys src/tools src/ubin src/usbin world/base world/info world/lib32 world/manpages The following components of FreeBSD do not seem to be installed: kernel/generic world/catpages world/dict world/doc world/games world/proflibs Does this look reasonable (y/n)? y At this point, freebsd-update will attempt to download all files required for the upgrade. In some cases, the user may be prompted with questions regarding what to install or how to proceed. When using a custom kernel, the above step will produce a warning similar to the following: WARNING: This system is running a "MYKERNEL" kernel, which is not a kernel configuration distributed as part of FreeBSD 9.0-RELEASE. This kernel will not be updated: you MUST update the kernel manually before running "/usr/sbin/freebsd-update install" This warning may be safely ignored at this point. The updated GENERIC kernel will be used as an intermediate step in the upgrade process. Once all the patches have been downloaded to the local system, they will be applied. This process may take a while, depending on the speed and workload of the machine. Configuration files will then be merged. The merging process requires some user intervention as a file may be merged or an editor may appear on screen for a manual merge. The results of every successful merge will be shown to the user as the process continues. A failed or ignored merge will cause the process to abort. Users may wish to make a backup of /etc and manually merge important files, such as master.passwd or group at a later time. The system is not being altered yet as all patching and merging is happening in another directory. Once all patches have been applied successfully, all configuration files have been merged and it seems the process will go smoothly, the changes can be committed to disk by the user using the following command: &prompt.root; freebsd-update install The kernel and kernel modules will be patched first. If the system is running with a custom kernel, use &man.nextboot.8; to set the kernel for the next boot to the updated /boot/GENERIC: &prompt.root; nextboot -k GENERIC Before rebooting with the GENERIC kernel, make sure it contains all the drivers required for the system to boot properly and connect to the network, if the machine being updated is accessed remotely. In particular, if the running custom kernel contains built-in functionality usually provided by kernel modules, make sure to temporarily load these modules into the GENERIC kernel using the /boot/loader.conf facility. It is recommended to disable non-essential services as well as any disk and network mounts until the upgrade process is complete. The machine should now be restarted with the updated kernel: &prompt.root; shutdown -r now Once the system has come back online, restart freebsd-update using the following command. Since the state of the process has been saved, freebsd-update will not start from the beginning, but will instead move on to the next phase and remove all old shared libraries and object files. &prompt.root; freebsd-update install Depending upon whether any library version numbers were bumped, there may only be two install phases instead of three. The upgrade is now complete. If this was a major version upgrade, reinstall all ports and packages as described in . Custom Kernels with &os; 9.X and Later Before using freebsd-update, ensure that a copy of the GENERIC kernel exists in /boot/GENERIC. If a custom kernel has only been built once, the kernel in /boot/kernel.old is the GENERIC kernel. Simply rename this directory to /boot/kernel. If a custom kernel has been built more than once or if it is unknown how many times the custom kernel has been built, obtain a copy of the GENERIC kernel that matches the current version of the operating system. If physical access to the system is available, a copy of the GENERIC kernel can be installed from the installation media: &prompt.root; mount /cdrom &prompt.root; cd /cdrom/usr/freebsd-dist &prompt.root; tar -C/ -xvf kernel.txz boot/kernel/kernel Alternately, the GENERIC kernel may be rebuilt and installed from source: &prompt.root; cd /usr/src &prompt.root; make kernel __MAKE_CONF=/dev/null SRCCONF=/dev/null For this kernel to be identified as the GENERIC kernel by freebsd-update, the GENERIC configuration file must not have been modified in any way. It is also suggested that the kernel is built without any other special options. Rebooting into the GENERIC kernel is not required as freebsd-update only needs /boot/GENERIC to exist. Upgrading Packages After a Major Version Upgrade Generally, installed applications will continue to work without problems after minor version upgrades. Major versions use different Application Binary Interfaces (ABIs), which will break most third-party applications. After a major version upgrade, all installed packages and ports need to be upgraded. Packages can be upgraded using pkg upgrade. To upgrade installed ports, use a utility such as ports-mgmt/portmaster. A forced upgrade of all installed packages will replace the packages with fresh versions from the repository even if the version number has not increased. This is required because of the ABI version change when upgrading between major versions of &os;. The forced upgrade can be accomplished by performing: &prompt.root; pkg-static upgrade -f A rebuild of all installed applications can be accomplished with this command: &prompt.root; portmaster -af This command will display the configuration screens for each application that has configurable options and wait for the user to interact with those screens. To prevent this behavior, and use only the default options, include in the above command. Once the software upgrades are complete, finish the upgrade process with a final call to freebsd-update in order to tie up all the loose ends in the upgrade process: &prompt.root; freebsd-update install If the GENERIC kernel was temporarily used, this is the time to build and install a new custom kernel using the instructions in . Reboot the machine into the new &os; version. The upgrade process is now complete. System State Comparison The state of the installed &os; version against a known good copy can be tested using freebsd-update IDS. This command evaluates the current version of system utilities, libraries, and configuration files and can be used as a built-in Intrusion Detection System (IDS). This command is not a replacement for a real IDS such as security/snort. As freebsd-update stores data on disk, the possibility of tampering is evident. While this possibility may be reduced using kern.securelevel and by storing the freebsd-update data on a read-only file system when not in use, a better solution would be to compare the system against a secure disk, such as a DVD or securely stored external USB disk device. An alternative method for providing IDS functionality using a built-in utility is described in To begin the comparison, specify the output file to save the results to: &prompt.root; freebsd-update IDS >> outfile.ids The system will now be inspected and a lengthy listing of files, along with the SHA256 hash values for both the known value in the release and the current installation, will be sent to the specified output file. The entries in the listing are extremely long, but the output format may be easily parsed. For instance, to obtain a list of all files which differ from those in the release, issue the following command: &prompt.root; cat outfile.ids | awk '{ print $1 }' | more /etc/master.passwd /etc/motd /etc/passwd /etc/pf.conf This sample output has been truncated as many more files exist. Some files have natural modifications. For example, /etc/passwd will be modified if users have been added to the system. Kernel modules may differ as freebsd-update may have updated them. To exclude specific files or directories, add them to the IDSIgnorePaths option in /etc/freebsd-update.conf. Updating the Documentation Set Updating and Upgrading Documentation Updating and Upgrading Documentation is an integral part of the &os; operating system. While an up-to-date version of the &os; documentation is always available on the &os; web site (http://www.freebsd.org/doc/), it can be handy to have an up-to-date, local copy of the &os; website, handbooks, FAQ, and articles. This section describes how to use either source or the &os; Ports Collection to keep a local copy of the &os; documentation up-to-date. For information on editing and submitting corrections to the documentation, refer to the &os; Documentation Project Primer for New Contributors (http://www.freebsd.org/doc/en_US.ISO8859-1/books/fdp-primer/). Updating Documentation from Source Rebuilding the &os; documentation from source requires a collection of tools which are not part of the &os; base system. The required tools, including svn, can be installed from the textproc/docproj package or port developed by the &os; Documentation Project. Once installed, use svn to - fetch a clean copy of the documentation source. Replace - https://svn0.us-west.FreeBSD.org - with the address of the closest geographic mirror from : + fetch a clean copy of the documentation source: - &prompt.root; svn checkout https://svn0.us-west.FreeBSD.org/doc/head /usr/doc + &prompt.root; svn checkout https://svn.FreeBSD.org/doc/head /usr/doc The initial download of the documentation sources may take a while. Let it run until it completes. Future updates of the documentation sources may be fetched by running: &prompt.root; svn update /usr/doc Once an up-to-date snapshot of the documentation sources has been fetched to /usr/doc, everything is ready for an update of the installed documentation. A full update of all available languages may be performed by typing: &prompt.root; cd /usr/doc &prompt.root; make install clean If an update of only a specific language is desired, make can be invoked in a language-specific subdirectory of /usr/doc: &prompt.root; cd /usr/doc/en_US.ISO8859-1 &prompt.root; make install clean An alternative way of updating the documentation is to run this command from /usr/doc or the desired language-specific subdirectory: &prompt.root; make update The output formats that will be installed may be specified by setting FORMATS: &prompt.root; cd /usr/doc &prompt.root; make FORMATS='html html-split' install clean Several options are available to ease the process of updating only parts of the documentation, or the build of specific translations. These options can be set either as system-wide options in /etc/make.conf, or as command-line options passed to make. The options include: DOC_LANG The list of languages and encodings to build and install, such as en_US.ISO8859-1 for English documentation. FORMATS A single format or a list of output formats to be built. Currently, html, html-split, txt, ps, and pdf are supported. DOCDIR Where to install the documentation. It defaults to /usr/share/doc. For more make variables supported as system-wide options in &os;, refer to &man.make.conf.5;. Updating Documentation from Ports Marc Fonvieille Based on the work of Updating and Upgrading documentation package Updating and Upgrading The previous section presented a method for updating the &os; documentation from sources. This section describes an alternative method which uses the Ports Collection and makes it possible to: Install pre-built packages of the documentation, without having to locally build anything or install the documentation toolchain. Build the documentation sources through the ports framework, making the checkout and build steps a bit easier. This method of updating the &os; documentation is supported by a set of documentation ports and packages which are updated by the &a.doceng; on a monthly basis. These are listed in the &os; Ports Collection, under the docs category (http://www.freshports.org/docs/). Organization of the documentation ports is as follows: The misc/freebsd-doc-en package or port installs all of the English documentation. The misc/freebsd-doc-all meta-package or port installs all documentation in all available languages. There is a package and port for each translation, such as misc/freebsd-doc-hu for the Hungarian documentation. When binary packages are used, the &os; documentation will be installed in all available formats for the given language. For example, the following command will install the latest package of the Hungarian documentation: &prompt.root; pkg install hu-freebsd-doc Packages use a format that differs from the corresponding port's name: lang-freebsd-doc, where lang is the short format of the language code, such as hu for Hungarian, or zh_cn for Simplified Chinese. To specify the format of the documentation, build the port instead of installing the package. For example, to build and install the English documentation: &prompt.root; cd /usr/ports/misc/freebsd-doc-en &prompt.root; make install clean The port provides a configuration menu where the format to build and install can be specified. By default, split HTML, similar to the format used on http://www.FreeBSD.org, and PDF are selected. Alternately, several make options can be specified when building a documentation port, including: WITH_HTML Builds the HTML format with a single HTML file per document. The formatted documentation is saved to a file called article.html, or book.html. WITH_PDF The formatted documentation is saved to a file called article.pdf or book.pdf. DOCBASE Specifies where to install the documentation. It defaults to /usr/local/share/doc/freebsd. This example uses variables to install the Hungarian documentation as a PDF in the specified directory: &prompt.root; cd /usr/ports/misc/freebsd-doc-hu &prompt.root; make -DWITH_PDF DOCBASE=share/doc/freebsd/hu install clean Documentation packages or ports can be updated using the instructions in . For example, the following command updates the installed Hungarian documentation using ports-mgmt/portmaster by using packages only: &prompt.root; portmaster -PP hu-freebsd-doc Tracking a Development Branch -CURRENT -STABLE &os; has two development branches: &os.current; and &os.stable;. This section provides an explanation of each branch and its intended audience, as well as how to keep a system up-to-date with each respective branch. Using &os.current; &os.current; is the bleeding edge of &os; development and &os.current; users are expected to have a high degree of technical skill. Less technical users who wish to track a development branch should track &os.stable; instead. &os.current; is the very latest source code for &os; and includes works in progress, experimental changes, and transitional mechanisms that might or might not be present in the next official release. While many &os; developers compile the &os.current; source code daily, there are short periods of time when the source may not be buildable. These problems are resolved as quickly as possible, but whether or not &os.current; brings disaster or new functionality can be a matter of when the source code was synced. &os.current; is made available for three primary interest groups: Members of the &os; community who are actively working on some part of the source tree. Members of the &os; community who are active testers. They are willing to spend time solving problems, making topical suggestions on changes and the general direction of &os;, and submitting patches. Users who wish to keep an eye on things, use the current source for reference purposes, or make the occasional comment or code contribution. &os.current; should not be considered a fast-track to getting new features before the next release as pre-release features are not yet fully tested and most likely contain bugs. It is not a quick way of getting bug fixes as any given commit is just as likely to introduce new bugs as to fix existing ones. &os.current; is not in any way officially supported. -CURRENT using To track &os.current;: Join the &a.current.name; and the &a.svn-src-head.name; lists. This is essential in order to see the comments that people are making about the current state of the system and to receive important bulletins about the current state of &os.current;. The &a.svn-src-head.name; list records the commit log entry for each change as it is made, along with any pertinent information on possible side effects. To join these lists, go to &a.mailman.lists.link;, click on the list to subscribe to, and follow the instructions. In order to track changes to the whole source tree, not just the changes to &os.current;, subscribe to the &a.svn-src-all.name; list. Synchronize with the &os.current; sources. Typically, svn is used to check out the -CURRENT code from the head branch of one of the Subversion mirror sites listed in . Users with very slow or limited Internet connectivity can instead use CTM as described in , but it is not as reliable as svn and svn is the recommended method for synchronizing source. Due to the size of the repository, some users choose to only synchronize the sections of source that interest them or which they are contributing patches to. However, users that plan to compile the operating system from source must download all of &os.current;, not just selected portions. Before compiling &os.current; -CURRENT compiling , read /usr/src/Makefile very carefully and follow the instructions in . Read the &a.current; and /usr/src/UPDATING to stay up-to-date on other bootstrapping procedures that sometimes become necessary on the road to the next release. Be active! &os.current; users are encouraged to submit their suggestions for enhancements or bug fixes. Suggestions with accompanying code are always welcome. Using &os.stable; &os.stable; is the development branch from which major releases are made. Changes go into this branch at a slower pace and with the general assumption that they have first been tested in &os.current;. This is still a development branch and, at any given time, the sources for &os.stable; may or may not be suitable for general use. It is simply another engineering development track, not a resource for end-users. Users who do not have the resources to perform testing should instead run the most recent release of &os;. Those interested in tracking or contributing to the &os; development process, especially as it relates to the next release of &os;, should consider following &os.stable;. While the &os.stable; branch should compile and run at all times, this cannot be guaranteed. Since more people run &os.stable; than &os.current;, it is inevitable that bugs and corner cases will sometimes be found in &os.stable; that were not apparent in &os.current;. For this reason, one should not blindly track &os.stable;. It is particularly important not to update any production servers to &os.stable; without thoroughly testing the code in a development or testing environment. To track &os.stable;: -STABLE using Join the &a.stable.name; list in order to stay informed of build dependencies that may appear in &os.stable; or any other issues requiring special attention. Developers will also make announcements in this mailing list when they are contemplating some controversial fix or update, giving the users a chance to respond if they have any issues to raise concerning the proposed change. Join the relevant svn list for the branch being tracked. For example, users tracking the 9-STABLE branch should join the &a.svn-src-stable-9.name; list. This list records the commit log entry for each change as it is made, along with any pertinent information on possible side effects. To join these lists, go to &a.mailman.lists.link;, click on the list to subscribe to, and follow the instructions. In order to track changes for the whole source tree, subscribe to &a.svn-src-all.name;. To install a new &os.stable; system, install the most recent &os.stable; release from the &os; mirror sites or use a monthly snapshot built from &os.stable;. Refer to www.freebsd.org/snapshots for more information about snapshots. To compile or upgrade to an existing &os; system to &os.stable;, use svn Subversion to check out the source for the desired branch. Branch names, such as stable/9, are listed at www.freebsd.org/releng. CTM () can be used if a reliable Internet connection is not available. Before compiling or upgrading to &os.stable; -STABLE compiling , read /usr/src/Makefile carefully and follow the instructions in . Read &a.stable; and /usr/src/UPDATING to keep up-to-date on other bootstrapping procedures that sometimes become necessary on the road to the next release. Synchronizing Source There are various methods for staying up-to-date with the &os; sources. This section compares the primary services, Subversion and CTM. While it is possible to update only parts of the source tree, the only supported update procedure is to update the entire tree and recompile all the programs that run in user space, such as those in /bin and /sbin, and kernel sources. Updating only part of the source tree, only the kernel, or only the userland programs will often result in problems ranging from compile errors to kernel panics or data corruption. Subversion Subversion uses the pull model of updating sources. The user, or a cron script, invokes the svn program which updates the local version of the source. Subversion is the preferred method for updating local source trees as updates are up-to-the-minute and the user controls when updates are downloaded. It is easy to restrict updates to specific files or directories and the requested updates are generated on the fly by the server. How to synchronize source using Subversion is described in . CTM CTM does not interactively compare the local sources with those on the master archive or otherwise pull them across. Instead, a script which identifies changes in files since its previous run is executed several times a day on the master CTM machine. Any detected changes are compressed, stamped with a sequence-number, and encoded for transmission over email in printable ASCII only. Once downloaded, these deltas can be run through ctm.rmail which will automatically decode, verify, and apply the changes to the user's copy of the sources. This process is more efficient than Subversion and places less strain on server resources since it is a push, rather than a pull, model. Instructions for using CTM to synchronize source can be found at . If a user inadvertently wipes out portions of the local archive, Subversion will detect and rebuild the damaged portions. CTM will not, and if a user deletes some portion of the source tree and does not have a backup, they will have to start from scratch from the most recent base delta and rebuild it all with CTM. Rebuilding World Rebuilding world Once the local source tree is synchronized against a particular version of &os; such as &os.stable; or &os.current;, the source tree can be used to rebuild the system. This process is known as rebuilding world. Before rebuilding world, be sure to perform the following tasks: Perform These Tasks <emphasis>Before</emphasis> Building World Backup all important data to another system or removable media, verify the integrity of the backup, and have a bootable installation media at hand. It cannot be stressed enough how important it is to make a backup of the system before rebuilding the system. While rebuilding world is an easy task, there will inevitably be times when mistakes in the source tree render the system unbootable. You will probably never have to use the backup, but it is better to be safe than sorry! mailing list Review the recent &a.stable.name; or &a.current.name; entries, depending upon the branch being tracked. Be aware of any known problems and which systems are affected. If a known issue affects the version of synchronized code, wait for an all clear announcement to be posted stating that the problem has been solved. Resynchronize the sources to ensure that the local version of source has the needed fix. Read /usr/src/UPDATING for any extra steps necessary for that version of the source. This file contains important information about potential problems and may specify the order to run certain commands. Many upgrades require specific additional steps such as renaming or deleting specific files prior to installing the new world. These will be listed at the end of this file where the currently recommended upgrade sequence is explicitly spelled out. If UPDATING contradicts any steps in this chapter, the instructions in UPDATING take precedence and should be followed. Do Not Use <command>make world</command> Some older documentation recommends using make world. However, that command skips some important steps and should only be used by experts. For almost all circumstances make world is the wrong thing to do, and the procedure described here should be used instead. Overview of Process The build world process assumes an upgrade from an older &os; version using the source of a newer version that was obtained using the instructions in . In &os;, the term world includes the kernel, core system binaries, libraries, programming files, and built-in compiler. The order in which these components are built and installed is important. For example, the old compiler might have a bug and not be able to compile the new kernel. Since the new kernel should be built with the new compiler, the new compiler must be built, but not necessarily installed, before the new kernel is built. The new world might rely on new kernel features, so the new kernel must be installed before the new world is installed. The old world might not run correctly on the new kernel, so the new world must be installed immediately upon installing the new kernel. Some configuration changes must be made before the new world is installed, but others might break the old world. Hence, two different configuration upgrade steps are used. For the most part, the update process only replaces or adds files and existing old files are not deleted. Since this can cause problems, /usr/src/UPDATING will indicate if any files need to be manually deleted and at which step to do so. These concerns have led to the recommended upgrade sequence described in the following procedure. It is a good idea to save the output from running make to a file. If something goes wrong, a copy of the error message can be posted to one of the &os; mailing lists. The easiest way to do this is to use script with a parameter that specifies the name of the file to save all output to. Do not save the output to /tmp as this directory may be cleared at next reboot. A better place to save the file is /var/tmp. Run this command immediately before rebuilding the world, and then type exit when the process has finished: &prompt.root; script /var/tmp/mw.out Script started, output file is /var/tmp/mw.out Overview of Build World Process The commands used in the build world process should be run in the order specified here. This section summarizes the function of each command. If the build world process has previously been run on this system, a copy of the previous build may still exist in /usr/obj. To speed up the new build world process, and possibly save some dependency headaches, remove this directory if it already exists: &prompt.root; chflags -R noschg /usr/obj/* &prompt.root; rm -rf /usr/obj Compile the new compiler and a few related tools, then use the new compiler to compile the rest of the new world. The result is saved to /usr/obj. &prompt.root; cd /usr/src &prompt.root; make buildworld Use the new compiler residing in /usr/obj to build the new kernel, in order to protect against compiler-kernel mismatches. This is necessary, as certain memory structures may have changed, and programs like ps and top will fail to work if the kernel and source code versions are not the same. &prompt.root; make buildkernel Install the new kernel and kernel modules, making it possible to boot with the newly updated kernel. If kern.securelevel has been raised above 1 and noschg or similar flags have been set on the kernel binary, drop the system into single-user mode first. Otherwise, this command can be run from multi-user mode without problems. See &man.init.8; for details about kern.securelevel and &man.chflags.1; for details about the various file flags. &prompt.root; make installkernel Drop the system into single-user mode in order to minimize problems from updating any binaries that are already running. It also minimizes any problems from running the old world on a new kernel. &prompt.root; shutdown now Once in single-user mode, run these commands if the system is formatted with UFS: &prompt.root; mount -u / &prompt.root; mount -a -t ufs &prompt.root; swapon -a If the system is instead formatted with ZFS, run these two commands. This example assumes a zpool name of zroot: &prompt.root; zfs set readonly=off zroot &prompt.root; zfs mount -a Optional: If a keyboard mapping other than the default US English is desired, it can be changed with &man.kbdmap.1;: &prompt.root; kbdmap Then, for either file system, if the CMOS clock is set to local time (this is true if the output of &man.date.1; does not show the correct time and zone), run: &prompt.root; adjkerntz -i Remaking the world will not update certain directories, such as /etc, /var and /usr, with new or changed configuration files. The next step is to perform some initial configuration file updates to /etc in preparation for the new world. The following command compares only those files that are essential for the success of installworld. For instance, this step may add new groups, system accounts, or startup scripts which have been added to &os; since the last update. This is necessary so that the installworld step will be able to use any new system accounts, groups, and scripts. Refer to for more detailed instructions about this command: &prompt.root; mergemaster -p Install the new world and system binaries from /usr/obj. &prompt.root; cd /usr/src &prompt.root; make installworld Update any remaining configuration files. &prompt.root; mergemaster -iF Delete any obsolete files. This is important as they may cause problems if left on the disk. &prompt.root; make delete-old A full reboot is now needed to load the new kernel and new world with the new configuration files. &prompt.root; reboot Make sure that all installed ports have first been rebuilt before old libraries are removed using the instructions in . When finished, remove any obsolete libraries to avoid conflicts with newer ones. For a more detailed description of this step, refer to . &prompt.root; make delete-old-libs single-user mode If the system can have a window of down-time, consider compiling the system in single-user mode instead of compiling the system in multi-user mode, and then dropping into single-user mode for the installation. Reinstalling the system touches a lot of important system files, all the standard system binaries, libraries, and include files. Changing these on a running system, particularly one with active users, is asking for trouble. Configuration Files make.conf This build world process uses several configuration files. The Makefile located in /usr/src describes how the programs that comprise &os; should be built and the order in which they should be built. The options available to make are described in &man.make.conf.5; and some common examples are included in /usr/share/examples/etc/make.conf. Any options which are added to /etc/make.conf will control the how make runs and builds programs. These options take effect every time make is used, including compiling applications from the Ports Collection, compiling custom C programs, or building the &os; operating system. Changes to some settings can have far-reaching and potentially surprising effects. Read the comments in both locations and keep in mind that the defaults have been chosen for a combination of performance and safety. src.conf How the operating system is built from source code is controlled by /etc/src.conf. Unlike /etc/make.conf, the contents of /etc/src.conf only take effect when the &os; operating system itself is being built. Descriptions of the many options available for this file are shown in &man.src.conf.5;. Be cautious about disabling seemingly unneeded kernel modules and build options. Sometimes there are unexpected or subtle interactions. Variables and Targets The general format for using make is as follows: &prompt.root; make -x -DVARIABLE target In this example, is an option passed to make. Refer to &man.make.1; for examples of the available options. To pass a variable, specify the variable name with . The behavior of the Makefile is controlled by variables. These can either be set in /etc/make.conf or they can be specified when using make. For example, this variable specifies that profiled libraries should not be built: &prompt.root; make -DNO_PROFILE target It corresponds with this setting in /etc/make.conf: NO_PROFILE= true # Avoid compiling profiled libraries The target tells make what to do and the Makefile defines the available targets. Some targets are used by the build process to break out the steps necessary to rebuild the system into a number of sub-steps. Having separate options is useful for two reasons. First, it allows for a build that does not affect any components of a running system. Because of this, buildworld can be safely run on a machine running in multi-user mode. It is still recommended that installworld be run in part in single-user mode, though. Secondly, it allows NFS mounts to be used to upgrade multiple machines on a network, as described in . It is possible to specify which will cause make to spawn several simultaneous processes. Since much of the compiling process is I/O-bound rather than CPU-bound, this is useful on both single CPU and multi-CPU machines. On a single-CPU machine, run the following command to have up to 4 processes running at any one time. Empirical evidence posted to the mailing lists shows this generally gives the best performance benefit. &prompt.root; make -j4 buildworld On a multi-CPU machine, try values between 6 and 10 to see how they speed things up. rebuilding world timings If any variables were specified to make buildworld, specify the same variables to make installworld. However, must never be used with installworld. For example, if this command was used: &prompt.root; make -DNO_PROFILE buildworld Install the results with: &prompt.root; make -DNO_PROFILE installworld Otherwise, the second command will try to install profiled libraries that were not built during the make buildworld phase. Merging Configuration Files Tom Rhodes Contributed by mergemaster &os; provides the &man.mergemaster.8; Bourne script to aid in determining the differences between the configuration files in /etc, and the configuration files in /usr/src/etc. This is the recommended solution for keeping the system configuration files up to date with those located in the source tree. Before using mergemaster, it is recommended to first copy the existing /etc somewhere safe. Include which does a recursive copy and which preserves times and the ownerships on files: &prompt.root; cp -Rp /etc /etc.old When run, mergemaster builds a temporary root environment, from / down, and populates it with various system configuration files. Those files are then compared to the ones currently installed in the system. Files that differ will be shown in &man.diff.1; format, with the sign representing added or modified lines, and representing lines that will be either removed completely or replaced with a new file. Refer to &man.diff.1; for more information about how file differences are shown. Next, mergemaster will display each file that differs, and present options to: delete the new file, referred to as the temporary file, install the temporary file in its unmodified state, merge the temporary file with the currently installed file, or view the results again. Choosing to delete the temporary file will tell mergemaster to keep the current file unchanged and to delete the new version. This option is not recommended. To get help at any time, type ? at the mergemaster prompt. If the user chooses to skip a file, it will be presented again after all other files have been dealt with. Choosing to install the unmodified temporary file will replace the current file with the new one. For most unmodified files, this is the best option. Choosing to merge the file will present a text editor, and the contents of both files. The files can be merged by reviewing both files side by side on the screen, and choosing parts from both to create a finished product. When the files are compared side by side, l selects the left contents and r selects contents from the right. The final output will be a file consisting of both parts, which can then be installed. This option is customarily used for files where settings have been modified by the user. Choosing to view the results again will redisplay the file differences. After mergemaster is done with the system files, it will prompt for other options. It may prompt to rebuild the password file and will finish up with an option to remove left-over temporary files. Deleting Obsolete Files and Libraries Anton Shterenlikht Based on notes provided by Deleting obsolete files and directories As a part of the &os; development lifecycle, files and their contents occasionally become obsolete. This may be because functionality is implemented elsewhere, the version number of the library has changed, or it was removed from the system entirely. These obsoleted files, libraries, and directories should be removed when updating the system. This ensures that the system is not cluttered with old files which take up unnecessary space on the storage and backup media. Additionally, if the old library has a security or stability issue, the system should be updated to the newer library to keep it safe and to prevent crashes caused by the old library. Files, directories, and libraries which are considered obsolete are listed in /usr/src/ObsoleteFiles.inc. The following instructions should be used to remove obsolete files during the system upgrade process. After the make installworld and the subsequent mergemaster have finished successfully, check for obsolete files and libraries: &prompt.root; cd /usr/src &prompt.root; make check-old If any obsolete files are found, they can be deleted using the following command: &prompt.root; make delete-old A prompt is displayed before deleting each obsolete file. To skip the prompt and let the system remove these files automatically, use BATCH_DELETE_OLD_FILES: &prompt.root; make -DBATCH_DELETE_OLD_FILES delete-old The same goal can be achieved by piping these commands through yes: &prompt.root; yes|make delete-old Warning Deleting obsolete files will break applications that still depend on those obsolete files. This is especially true for old libraries. In most cases, the programs, ports, or libraries that used the old library need to be recompiled before make delete-old-libs is executed. Utilities for checking shared library dependencies include sysutils/libchk and sysutils/bsdadminscripts. Obsolete shared libraries can conflict with newer libraries, causing messages like these: /usr/bin/ld: warning: libz.so.4, needed by /usr/local/lib/libtiff.so, may conflict with libz.so.5 /usr/bin/ld: warning: librpcsvc.so.4, needed by /usr/local/lib/libXext.so, may conflict with librpcsvc.so.5 To solve these problems, determine which port installed the library: &prompt.root; pkg which /usr/local/lib/libtiff.so /usr/local/lib/libtiff.so was installed by package tiff-3.9.4 &prompt.root; pkg which /usr/local/lib/libXext.so /usr/local/lib/libXext.so was installed by package libXext-1.1.1,1 Then deinstall, rebuild, and reinstall the port. To automate this process, ports-mgmt/portmaster can be used. After all ports are rebuilt and no longer use the old libraries, delete the old libraries using the following command: &prompt.root; make delete-old-libs If something goes wrong, it is easy to rebuild a particular piece of the system. For example, if /etc/magic was accidentally deleted as part of the upgrade or merge of /etc, file will stop working. To fix this, run: &prompt.root; cd /usr/src/usr.bin/file &prompt.root; make all install Common Questions Do I need to re-make the world for every change? It depends upon the nature of the change. For example, if svn only shows the following files as being updated: src/games/cribbage/instr.c src/games/sail/pl_main.c src/release/sysinstall/config.c src/release/sysinstall/media.c src/share/mk/bsd.port.mk it probably is not worth rebuilding the entire world. Instead, go into the appropriate sub-directories and run make all install. But if something major changes, such as src/lib/libc/stdlib, consider rebuilding world. Some users rebuild world every fortnight and let changes accumulate over that fortnight. Others only re-make those things that have changed and are careful to spot all the dependencies. It all depends on how often a user wants to upgrade and whether they are tracking &os.stable; or &os.current;. What would cause a compile to fail with lots of signal 11 signal 11 (or other signal number) errors? This normally indicates a hardware problem. Building world is an effective way to stress test hardware, especially memory. A sure indicator of a hardware issue is when make is restarted and it dies at a different point in the process. To resolve this error, swap out the components in the machine, starting with RAM, to determine which component is failing. Can /usr/obj be removed when finished? This directory contains all the object files that were produced during the compilation phase. Normally, one of the first steps in the make buildworld process is to remove this directory and start afresh. Keeping /usr/obj around when finished makes little sense, and its removal frees up a approximately 2GB of disk space. Can interrupted builds be resumed? This depends on how far into the process the problem occurs. In general, make buildworld builds new copies of essential tools and the system libraries. These tools and libraries are then installed, used to rebuild themselves, and are installed again. The rest of the system is then rebuilt with the new system tools. During the last stage, it is fairly safe to run these commands as they will not undo the work of the previous make buildworld: &prompt.root; cd /usr/src &prompt.root; make -DNO_CLEAN all If this message appears: -------------------------------------------------------------- Building everything.. -------------------------------------------------------------- in the make buildworld output, it is probably fairly safe to do so. If that message is not displayed, it is always better to be safe than sorry and to restart the build from scratch. Is it possible to speed up making the world? Several actions can speed up the build world process. For example, the entire process can be run from single-user mode. However, this will prevent users from having access to the system until the process is complete. Careful file system design or the use of ZFS datasets can make a difference. Consider putting /usr/src and /usr/obj on separate file systems. If possible, place the file systems on separate disks on separate disk controllers. When mounting /usr/src, use which prevents the file system from recording the file access time. If /usr/src is not on its own file system, consider remounting /usr with . The file system holding /usr/obj can be mounted or remounted with so that disk writes happen asynchronously. The write completes immediately, and the data is written to the disk a few seconds later. This allows writes to be clustered together, and can provide a dramatic performance boost. Keep in mind that this option makes the file system more fragile. With this option, there is an increased chance that, should power fail, the file system will be in an unrecoverable state when the machine restarts. If /usr/obj is the only directory on this file system, this is not a problem. If you have other, valuable data on the same file system, ensure that there are verified backups before enabling this option. Turn off profiling by setting NO_PROFILE=true in /etc/make.conf. Pass to &man.make.1; to run multiple processes in parallel. This usually helps on both single- and multi-processor machines. What if something goes wrong? First, make absolutely sure that the environment has no extraneous cruft from earlier builds: &prompt.root; chflags -R noschg /usr/obj/usr &prompt.root; rm -rf /usr/obj/usr &prompt.root; cd /usr/src &prompt.root; make cleandir &prompt.root; make cleandir Yes, make cleandir really should be run twice. Then, restart the whole process, starting with make buildworld. If problems persist, send the error and the output of uname -a to &a.questions;. Be prepared to answer other questions about the setup! Tracking for Multiple Machines Mike Meyer Contributed by NFS installing multiple machines When multiple machines need to track the same source tree, it is a waste of disk space, network bandwidth, and CPU cycles to have each system download the sources and rebuild everything. The solution is to have one machine do most of the work, while the rest of the machines mount that work via NFS. This section outlines a method of doing so. For more information about using NFS, refer to . First, identify a set of machines which will run the same set of binaries, known as a build set. Each machine can have a custom kernel, but will run the same userland binaries. From that set, choose a machine to be the build machine that the world and kernel are built on. Ideally, this is a fast machine that has sufficient spare CPU to run make buildworld and make buildkernel. Select a machine to be the test machine, which will test software updates before they are put into production. This must be a machine that can afford to be down for an extended period of time. It can be the build machine, but need not be. All the machines in this build set need to mount /usr/obj and /usr/src from the build machine via NFS. For multiple build sets, /usr/src should be on one build machine, and NFS mounted on the rest. Ensure that /etc/make.conf and /etc/src.conf on all the machines in the build set agree with the build machine. That means that the build machine must build all the parts of the base system that any machine in the build set is going to install. Also, each build machine should have its kernel name set with KERNCONF in /etc/make.conf, and the build machine should list them all in its KERNCONF, listing its own kernel first. The build machine must have the kernel configuration files for each machine in its /usr/src/sys/arch/conf. On the build machine, build the kernel and world as described in , but do not install anything on the build machine. Instead, install the built kernel on the test machine. On the test machine, mount /usr/src and /usr/obj via NFS. Then, run shutdown now to go to single-user mode in order to install the new kernel and world and run mergemaster as usual. When done, reboot to return to normal multi-user operations. After verifying that everything on the test machine is working properly, use the same procedure to install the new software on each of the other machines in the build set. The same methodology can be used for the ports tree. The first step is to share /usr/ports via NFS to all the machines in the build set. To configure /etc/make.conf to share distfiles, set DISTDIR to a common shared directory that is writable by whichever user root is mapped to by the NFS mount. Each machine should set WRKDIRPREFIX to a local build directory, if ports are to be built locally. Alternately, if the build system is to build and distribute packages to the machines in the build set, set PACKAGES on the build system to a directory similar to DISTDIR. Index: head/en_US.ISO8859-1/books/handbook/ports/chapter.xml =================================================================== --- head/en_US.ISO8859-1/books/handbook/ports/chapter.xml (revision 47723) +++ head/en_US.ISO8859-1/books/handbook/ports/chapter.xml (revision 47724) @@ -1,1900 +1,1896 @@ Installing Applications: Packages and Ports Synopsis ports packages &os; is bundled with a rich collection of system tools as part of the base system. In addition, &os; provides two complementary technologies for installing third-party software: the &os; Ports Collection, for installing from source, and packages, for installing from pre-built binaries. Either method may be used to install software from local media or from the network. After reading this chapter, you will know: The difference between binary packages and ports. How to find third-party software that has been ported to &os;. How to manage binary packages using pkg. How to build third-party software from source using the Ports Collection. How to find the files installed with the application for post-installation configuration. What to do if a software installation fails. Overview of Software Installation The typical steps for installing third-party software on a &unix; system include: Find and download the software, which might be distributed in source code format or as a binary. Unpack the software from its distribution format. This is typically a tarball compressed with &man.compress.1;, &man.gzip.1;, or &man.bzip2.1;. Locate the documentation in INSTALL, README or some file in a doc/ subdirectory and read up on how to install the software. If the software was distributed in source format, compile it. This may involve editing a Makefile or running a configure script. Test and install the software. If the software package was not deliberately ported, or tested to work, on &os;, the source code may need editing in order for it to install and run properly. At the time of this writing, over &os.numports; third-party applications have been ported to &os;. A &os; package contains pre-compiled copies of all the commands for an application, as well as any configuration files and documentation. A package can be manipulated with the pkg commands, such as pkg install. A &os; port is a collection of files designed to automate the process of compiling an application from source code. The files that comprise a port contain all the necessary information to automatically download, extract, patch, compile, and install the application. The ports system can also be used to generate packages which can be manipulated with the &os; package management commands. Both packages and ports understand dependencies. If a package or port is used to install an application and a dependent library is not already installed, the library will automatically be installed first. While the two technologies are similar, packages and ports each have their own strengths. Select the technology that meets your requirements for installing a particular application. Package Benefits A compressed package tarball is typically smaller than the compressed tarball containing the source code for the application. Packages do not require compilation time. For large applications, such as Mozilla, KDE, or GNOME, this can be important on a slow system. Packages do not require any understanding of the process involved in compiling software on &os;. Port Benefits Packages are normally compiled with conservative options because they have to run on the maximum number of systems. By compiling from the port, one can change the compilation options. Some applications have compile-time options relating to which features are installed. For example, Apache can be configured with a wide variety of different built-in options. In some cases, multiple packages will exist for the same application to specify certain settings. For example, Ghostscript is available as a ghostscript package and a ghostscript-nox11 package, depending on whether or not Xorg is installed. Creating multiple packages rapidly becomes impossible if an application has more than one or two different compile-time options. The licensing conditions of some software forbid binary distribution. Such software must be distributed as source code which must be compiled by the end-user. Some people do not trust binary distributions or prefer to read through source code in order to look for potential problems. Source code is needed in order to apply custom patches. To keep track of updated ports, subscribe to the &a.ports; and the &a.ports-bugs;. Before installing any application, check for security issues related to the application or type pkg audit -F to check all installed applications for known vulnerabilities. The remainder of this chapter explains how to use packages and ports to install and manage third-party software on &os;. Finding Software &os;'s list of available applications is growing all the time. There are a number of ways to find software to install: The &os; web site maintains an up-to-date searchable list of all the available applications, at http://www.FreeBSD.org/ports/. The ports can be searched by application name or by software category. FreshPorts Dan Langille maintains FreshPorts.org which provides a comprehensive search utility and also tracks changes to the applications in the Ports Collection. Registered users can create a customized watch list in order to receive an automated email when their watched ports are updated. SourceForge If finding a particular application becomes challenging, try searching a site like SourceForge.net or GitHub.com then check back at the &os; site to see if the application has been ported. pkg search To search the binary package repository for an application: &prompt.root; pkg search subversion git-subversion-1.9.2 java-subversion-1.8.8_2 p5-subversion-1.8.8_2 py27-hgsubversion-1.6 py27-subversion-1.8.8_2 ruby-subversion-1.8.8_2 subversion-1.8.8_2 subversion-book-4515 subversion-static-1.8.8_2 subversion16-1.6.23_4 subversion17-1.7.16_2 Package names include the version number and in case of ports based on python, the version number of the version of python the package was built with. Some ports also have multiple versions available. In case of subversion there are different versions available, as well as different compile options. In this case, the staticly linked version of subversion. When indicating which package to install, it is best to specify the application by the port origin, which is the path in the ports tree. Repeat the pkg search with to list the origin of each package: &prompt.root; pkg search -o subversion devel/git-subversion java/java-subversion devel/p5-subversion devel/py-hgsubversion devel/py-subversion devel/ruby-subversion devel/subversion16 devel/subversion17 devel/subversion devel/subversion-book devel/subversion-static Searching by shell globs, regular expressions, exact match, by description, or any other field in the repository database is also supported by pkg search. After installing ports-mgmt/pkg or ports-mgmt/pkg-devel, see &man.pkg-search.8; for more details. If the Ports Collection is already installed, there are several methods to query the local version of the ports tree. To find out which category a port is in, type whereis file, where file is the program to be installed: &prompt.root; whereis lsof lsof: /usr/ports/sysutils/lsof Alternately, an &man.echo.1; statement can be used: &prompt.root; echo /usr/ports/*/*lsof* /usr/ports/sysutils/lsof Note that this will also return any matched files downloaded into the /usr/ports/distfiles directory. Another way to find software is by using the Ports Collection's built-in search mechanism. To use the search feature, cd to /usr/ports then run make search name=program-name where program-name is the name of the software. For example, to search for lsof: &prompt.root; cd /usr/ports &prompt.root; make search name=lsof Port: lsof-4.88.d,8 Path: /usr/ports/sysutils/lsof Info: Lists information about open files (similar to fstat(1)) Maint: ler@lerctr.org Index: sysutils B-deps: R-deps: The built-in search mechanism uses a file of index information. If a message indicates that the INDEX is required, run make fetchindex to download the current index file. With the INDEX present, make search will be able to perform the requested search. The Path: line indicates where to find the port. To receive less information, use the quicksearch feature: &prompt.root; cd /usr/ports &prompt.root; make quicksearch name=lsof Port: lsof-4.88.d,8 Path: /usr/ports/sysutils/lsof Info: Lists information about open files (similar to fstat(1)) For more in-depth searching, use make search key=string or make quicksearch key=string, where string is some text to search for. The text can be in comments, descriptions, or dependencies in order to find ports which relate to a particular subject when the name of the program is unknown. When using search or quicksearch, the search string is case-insensitive. Searching for LSOF will yield the same results as searching for lsof. Using <application>pkg</application> for Binary Package Management pkg is the next generation replacement for the traditional &os; package management tools, offering many features that make dealing with binary packages faster and easier. pkg is not a replacement for port management tools like ports-mgmt/portmaster or ports-mgmt/portupgrade. These tools can be used to install third-party software from both binary packages and the Ports Collection, while pkg installs only binary packages. Getting Started with <application>pkg</application> &os; includes a bootstrap utility which can be used to download and install pkg, along with its manual pages. To bootstrap the system, run: &prompt.root; /usr/sbin/pkg For earlier &os; versions, pkg must instead be installed from the Ports Collection or as a binary package. To install the port, run: &prompt.root; cd /usr/ports/ports-mgmt/pkg &prompt.root; make &prompt.root; make install clean When upgrading an existing system that originally used the older package system, the database must be converted to the new format, so that the new tools are aware of the already installed packages. Once pkg has been installed, the package database must be converted from the traditional format to the new format by running this command: &prompt.root; pkg2ng This step is not required for new installations that do not yet have any third-party software installed. This step is not reversible. Once the package database has been converted to the pkg format, the traditional pkg_* tools should no longer be used. The package database conversion may emit errors as the contents are converted to the new version. Generally, these errors can be safely ignored. However, a list of third-party software that was not successfully converted will be listed after pkg2ng has finished and these applications must be manually reinstalled. To ensure that the &os; Ports Collection registers new software with pkg, and not the traditional packages format, &os; versions earlier than 10.X require this line in /etc/make.conf: WITH_PKGNG= yes By default pkg uses the &os; package mirrors. For information about building a custom package repository, see Additional pkg configuration options are described in &man.pkg.conf.5;. Usage information for pkg is available in the &man.pkg.8; manpage or by running pkg without additional arguments. Each pkg command argument is documented in a command-specific manual page. To read the manual page for pkg install, for example, run either of these commands: &prompt.root; pkg help install &prompt.root; man pkg-install The rest of this section demonstrates common binary package management tasks which can be performed using pkg. Each demonstrated command provides many switches to customize its use. Refer to a command's help or man page for details and more examples. Obtaining Information About Installed Packages Information about the packages installed on a system can be viewed by running pkg info which, when run without any switches, will list the package version for either all installed packages or the specified package. For example, to see which version of pkg is installed, run: &prompt.root; pkg info pkg pkg-1.1.4_1 Installing and Removing Packages To install a binary package use the following command, where packagename is the name of the package to install: &prompt.root; pkg install packagename This command uses repository data to determine which version of the software to install and if it has any uninstalled dependencies. For example, to install curl: &prompt.root; pkg install curl Updating repository catalogue /usr/local/tmp/All/curl-7.31.0_1.txz 100% of 1181 kB 1380 kBps 00m01s /usr/local/tmp/All/ca_root_nss-3.15.1_1.txz 100% of 288 kB 1700 kBps 00m00s Updating repository catalogue The following 2 packages will be installed: Installing ca_root_nss: 3.15.1_1 Installing curl: 7.31.0_1 The installation will require 3 MB more space 0 B to be downloaded Proceed with installing packages [y/N]: y Checking integrity... done [1/2] Installing ca_root_nss-3.15.5_1... done [2/2] Installing curl-7.31.0_1... done Cleaning up cache files...Done The new package and any additional packages that were installed as dependencies can be seen in the installed packages list: &prompt.root; pkg info ca_root_nss-3.15.5_1 The root certificate bundle from the Mozilla Project curl-7.31.0_1 Non-interactive tool to get files from FTP, GOPHER, HTTP(S) servers pkg-1.1.4_6 New generation package manager Packages that are no longer needed can be removed with pkg delete. For example: &prompt.root; pkg delete curl The following packages will be deleted: curl-7.31.0_1 The deletion will free 3 MB Proceed with deleting packages [y/N]: y [1/1] Deleting curl-7.31.0_1... done Upgrading Installed Packages Installed packages can be upgraded to their latest versions by running: &prompt.root; pkg upgrade This command will compare the installed versions with those available in the repository catalogue and upgrade them from the repository. Auditing Installed Packages Occasionally, software vulnerabilities may be discovered in third-party applications. To address this, pkg includes a built-in auditing mechanism. To determine if there are any known vulnerabilities for the software installed on the system, run: &prompt.root; pkg audit -F Automatically Removing Leaf Dependencies Removing a package may leave behind dependencies which are no longer required. Unneeded packages that were installed as dependencies can be automatically detected and removed using: &prompt.root; pkg autoremove Packages to be autoremoved: ca_root_nss-3.13.5 The autoremoval will free 723 kB Proceed with autoremoval of packages [y/N]: y Deinstalling ca_root_nss-3.15.1_1... done Restoring the Package Database Unlike the traditional package management system, pkg includes its own package database backup mechanism. This functionality is enabled by default. To disable the periodic script from backing up the package database, set daily_backup_pkgdb_enable="NO" in &man.periodic.conf.5;. To restore the contents of a previous package database backup, run the following command replacing /path/to/pkg.sql with the location of the backup: &prompt.root; pkg backup -r /path/to/pkg.sql If restoring a backup taken by the periodic script, it must be decompressed prior to being restored. To run a manual backup of the pkg database, run the following command, replacing /path/to/pkg.sql with a suitable file name and location: &prompt.root; pkg backup -d /path/to/pkg.sql Removing Stale Packages By default, pkg stores binary packages in a cache directory defined by PKG_CACHEDIR in &man.pkg.conf.5;. Only copies of the latest installed packages are kept. Older versions of pkg kept all previous packages. To remove these outdated binary packages, run: &prompt.root; pkg clean The entire cache may be cleared by running: &prompt.root; pkg clean -a Modifying Package Metadata Software within the &os; Ports Collection can undergo major version number changes. To address this, pkg has a built-in command to update package origins. This can be useful, for example, if lang/php5 is renamed to lang/php53 so that lang/php5 can now represent version 5.4. To change the package origin for the above example, run: &prompt.root; pkg set -o lang/php5:lang/php53 As another example, to update lang/ruby18 to lang/ruby19, run: &prompt.root; pkg set -o lang/ruby18:lang/ruby19 As a final example, to change the origin of the libglut shared libraries from graphics/libglut to graphics/freeglut, run: &prompt.root; pkg set -o graphics/libglut:graphics/freeglut When changing package origins, it is important to reinstall packages that are dependent on the package with the modified origin. To force a reinstallation of dependent packages, run: &prompt.root; pkg install -Rf graphics/freeglut Using the Ports Collection The Ports Collection is a set of Makefiles, patches, and description files stored in /usr/ports. This set of files is used to compile and install applications on &os;. Before an application can be compiled using a port, the Ports Collection must first be installed. If it was not installed during the installation of &os;, use one of the following methods to install it: Portsnap Method The base system of &os; includes Portsnap. This is a fast and user-friendly tool for retrieving the Ports Collection and is the recommended choice for most users. This utility connects to a &os; site, verifies the secure key, and downloads a new copy of the Ports Collection. The key is used to verify the integrity of all downloaded files. To download a compressed snapshot of the Ports Collection into /var/db/portsnap: &prompt.root; portsnap fetch When running Portsnap for the first time, extract the snapshot into /usr/ports: &prompt.root; portsnap extract After the first use of Portsnap has been completed as shown above, /usr/ports can be updated as needed by running: &prompt.root; portsnap fetch &prompt.root; portsnap update When using fetch, the extract or the update operation may be run consecutively, like so: &prompt.root; portsnap fetch update Subversion Method If more control over the ports tree is needed or if local changes need to be maintained, Subversion can be used to obtain the Ports Collection. Refer to the Subversion Primer for a detailed description of Subversion. Subversion must be installed before it can be used to check out the ports tree. If a copy of the ports tree is already present, install Subversion like this: &prompt.root; cd /usr/ports/devel/subversion &prompt.root; make install clean If the ports tree is not available, or pkg is being used to manage packages, Subversion can be installed as a package: &prompt.root; pkg install subversion - Check out a copy of the ports tree. For better - performance, replace - svn0.us-east.FreeBSD.org with a - Subversion - mirror close to your geographic location: + Check out a copy of the ports tree: - &prompt.root; svn checkout https://svn0.us-east.FreeBSD.org/ports/head /usr/ports + &prompt.root; svn checkout https://svn.FreeBSD.org/ports/head /usr/ports As needed, update /usr/ports after the initial Subversion checkout: &prompt.root; svn update /usr/ports The Ports Collection installs a series of directories representing software categories with each category having a subdirectory for each application. Each subdirectory, also referred to as a ports skeleton, contains a set of files that tell &os; how to compile and install that program. Each port skeleton includes these files and directories: Makefile: contains statements that specify how the application should be compiled and where its components should be installed. distinfo: contains the names and checksums of the files that must be downloaded to build the port. files/: this directory contains any patches needed for the program to compile and install on &os;. This directory may also contain other files used to build the port. pkg-descr: provides a more detailed description of the program. pkg-plist: a list of all the files that will be installed by the port. It also tells the ports system which files to remove upon deinstallation. Some ports include pkg-message or other files to handle special situations. For more details on these files, and on ports in general, refer to the &os; Porter's Handbook. The port does not include the actual source code, also known as a distfile. The extract portion of building a port will automatically save the downloaded source to /usr/ports/distfiles. Installing Ports ports installing This section provides basic instructions on using the Ports Collection to install or remove software. The detailed description of available make targets and environment variables is available in &man.ports.7;. Before compiling any port, be sure to update the Ports Collection as described in the previous section. Since the installation of any third-party software can introduce security vulnerabilities, it is recommended to first check for known security issues related to the port. Alternately, run pkg audit -F before installing a new port. This command can be configured to automatically perform a security audit and an update of the vulnerability database during the daily security system check. For more information, refer to &man.pkg-audit.8; and &man.periodic.8;. Using the Ports Collection assumes a working Internet connection. It also requires superuser privilege. To compile and install the port, change to the directory of the port to be installed, then type make install at the prompt. Messages will indicate the progress: &prompt.root; cd /usr/ports/sysutils/lsof &prompt.root; make install >> lsof_4.88D.freebsd.tar.gz doesn't seem to exist in /usr/ports/distfiles/. >> Attempting to fetch from ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/. ===> Extracting for lsof-4.88 ... [extraction output snipped] ... >> Checksum OK for lsof_4.88D.freebsd.tar.gz. ===> Patching for lsof-4.88.d,8 ===> Applying FreeBSD patches for lsof-4.88.d,8 ===> Configuring for lsof-4.88.d,8 ... [configure output snipped] ... ===> Building for lsof-4.88.d,8 ... [compilation output snipped] ... ===> Installing for lsof-4.88.d,8 ... [installation output snipped] ... ===> Generating temporary packing list ===> Compressing manual pages for lsof-4.88.d,8 ===> Registering installation for lsof-4.88.d,8 ===> SECURITY NOTE: This port has installed the following binaries which execute with increased privileges. /usr/local/sbin/lsof &prompt.root; Since lsof is a program that runs with increased privileges, a security warning is displayed as it is installed. Once the installation is complete, the prompt will be returned. Some shells keep a cache of the commands that are available in the directories listed in the PATH environment variable, to speed up lookup operations for the executable file of these commands. Users of the tcsh shell should type rehash so that a newly installed command can be used without specifying its full path. Use hash -r instead for the sh shell. Refer to the documentation for the shell for more information. During installation, a working subdirectory is created which contains all the temporary files used during compilation. Removing this directory saves disk space and minimizes the chance of problems later when upgrading to the newer version of the port: &prompt.root; make clean ===> Cleaning for lsof-88.d,8 &prompt.root; To save this extra step, instead use make install clean when compiling the port. Customizing Ports Installation Some ports provide build options which can be used to enable or disable application components, provide security options, or allow for other customizations. Examples include www/firefox, security/gpgme, and mail/sylpheed-claws. If the port depends upon other ports which have configurable options, it may pause several times for user interaction as the default behavior is to prompt the user to select options from a menu. To avoid this, run make config-recursive within the port skeleton to do this configuration in one batch. Then, run make install [clean] to compile and install the port. When using config-recursive, the list of ports to configure are gathered by the all-depends-list target. It is recommended to run make config-recursive until all dependent ports options have been defined, and ports options screens no longer appear, to be certain that all dependency options have been configured. There are several ways to revisit a port's build options menu in order to add, remove, or change these options after a port has been built. One method is to cd into the directory containing the port and type make config. Another option is to use make showconfig. Another option is to execute make rmconfig which will remove all selected options and allow you to start over. All of these options, and others, are explained in great detail in &man.ports.7;. The ports system uses &man.fetch.1; to download the source files, which supports various environment variables. The FTP_PASSIVE_MODE, FTP_PROXY, and FTP_PASSWORD variables may need to be set if the &os; system is behind a firewall or FTP/HTTP proxy. See &man.fetch.3; for the complete list of supported variables. For users who cannot be connected to the Internet all the time, make fetch can be run within /usr/ports, to fetch all distfiles, or within a category, such as /usr/ports/net, or within the specific port skeleton. Note that if a port has any dependencies, running this command in a category or ports skeleton will not fetch the distfiles of ports from another category. Instead, use make fetch-recursive to also fetch the distfiles for all the dependencies of a port. In rare cases, such as when an organization has a local distfiles repository, the MASTER_SITES variable can be used to override the download locations specified in the Makefile. When using, specify the alternate location: &prompt.root; cd /usr/ports/directory &prompt.root; make MASTER_SITE_OVERRIDE= \ ftp://ftp.organization.org/pub/FreeBSD/ports/distfiles/ fetch The WRKDIRPREFIX and PREFIX variables can override the default working and target directories. For example: &prompt.root; make WRKDIRPREFIX=/usr/home/example/ports install will compile the port in /usr/home/example/ports and install everything under /usr/local. &prompt.root; make PREFIX=/usr/home/example/local install will compile the port in /usr/ports and install it in /usr/home/example/local. And: &prompt.root; make WRKDIRPREFIX=../ports PREFIX=../local install will combine the two. These can also be set as environmental variables. Refer to the manual page for your shell for instructions on how to set an environmental variable. Removing Installed Ports ports removing Installed ports can be uninstalled using pkg delete. Examples for using this command can be found in the &man.pkg-delete.8; manpage. Alternately, make deinstall can be run in the port's directory: &prompt.root; cd /usr/ports/sysutils/lsof make deinstall ===> Deinstalling for sysutils/lsof ===> Deinstalling Deinstallation has been requested for the following 1 packages: lsof-4.88.d,8 The deinstallation will free 229 kB [1/1] Deleting lsof-4.88.d,8... done It is recommended to read the messages as the port is uninstalled. If the port has any applications that depend upon it, this information will be displayed but the uninstallation will proceed. In such cases, it may be better to reinstall the application in order to prevent broken dependencies. Upgrading Ports ports upgrading Over time, newer versions of software become available in the Ports Collection. This section describes how to determine which software can be upgraded and how to perform the upgrade. To determine if newer versions of installed ports are available, ensure that the latest version of the ports tree is installed, using the updating command described in either or . On &os; 10 and later, or if the system has been converted to pkg, the following command will list the installed ports which are out of date: &prompt.root; pkg version -l "<" For &os; 9.X and lower, the following command will list the installed ports that are out of date: &prompt.root; pkg_version -l "<" Before attempting an upgrade, read /usr/ports/UPDATING from the top of the file to the date closest to the last time ports were upgraded or the system was installed. This file describes various issues and additional steps users may encounter and need to perform when updating a port, including such things as file format changes, changes in locations of configuration files, or any incompatibilities with previous versions. Make note of any instructions which match any of the ports that need upgrading and follow these instructions when performing the upgrade. To perform the actual upgrade, use either Portmaster or Portupgrade. Upgrading Ports Using <application>Portmaster</application> portmaster The ports-mgmt/portmaster package or port is the recommended tool for upgrading installed ports as it is designed to use the tools installed with &os; without depending upon other ports. It uses the information in /var/db/pkg/ to determine which ports to upgrade. To install this utility as a port: &prompt.root; cd /usr/ports/ports-mgmt/portmaster &prompt.root; make install clean Portmaster defines four categories of ports: Root port: has no dependencies and is not a dependency of any other ports. Trunk port: has no dependencies, but other ports depend upon it. Branch port: has dependencies and other ports depend upon it. Leaf port: has dependencies but no other ports depend upon it. To list these categories and search for updates: &prompt.root; portmaster -L ===>>> Root ports (No dependencies, not depended on) ===>>> ispell-3.2.06_18 ===>>> screen-4.0.3 ===>>> New version available: screen-4.0.3_1 ===>>> tcpflow-0.21_1 ===>>> 7 root ports ... ===>>> Branch ports (Have dependencies, are depended on) ===>>> apache22-2.2.3 ===>>> New version available: apache22-2.2.8 ... ===>>> Leaf ports (Have dependencies, not depended on) ===>>> automake-1.9.6_2 ===>>> bash-3.1.17 ===>>> New version available: bash-3.2.33 ... ===>>> 32 leaf ports ===>>> 137 total installed ports ===>>> 83 have new versions available This command is used to upgrade all outdated ports: &prompt.root; portmaster -a By default, Portmaster will make a backup package before deleting the existing port. If the installation of the new version is successful, Portmaster will delete the backup. Using will instruct Portmaster not to automatically delete the backup. Adding will start Portmaster in interactive mode, prompting for confirmation before upgrading each port. Many other options are available. Read through the manual page for portmaster(8) for details regarding their usage. If errors are encountered during the upgrade process, add to upgrade and rebuild all ports: &prompt.root; portmaster -af Portmaster can also be used to install new ports on the system, upgrading all dependencies before building and installing the new port. To use this function, specify the location of the port in the Ports Collection: &prompt.root; portmaster shells/bash Upgrading Ports Using Portupgrade portupgrade Another utility that can be used to upgrade ports is Portupgrade, which is available as the ports-mgmt/portupgrade package or port. This utility installs a suite of applications which can be used to manage ports. However, it is dependent upon Ruby. To install the port: &prompt.root; cd /usr/ports/ports-mgmt/portupgrade &prompt.root; make install clean Before performing an upgrade using this utility, it is recommended to scan the list of installed ports using pkgdb -F and to fix all the inconsistencies it reports. To upgrade all the outdated ports installed on the system, use portupgrade -a. Alternately, include to be asked for confirmation of every individual upgrade: &prompt.root; portupgrade -ai To upgrade only a specified application instead of all available ports, use portupgrade pkgname. It is very important to include to first upgrade all the ports required by the given application: &prompt.root; portupgrade -R firefox If is included, Portupgrade searches for available packages in the local directories listed in PKG_PATH. If none are available locally, it then fetches packages from a remote site. If packages can not be found locally or fetched remotely, Portupgrade will use ports. To avoid using ports entirely, specify . This last set of options tells Portupgrade to abort if no packages are available: &prompt.root; portupgrade -PP gnome3 To just fetch the port distfiles, or packages, if is specified, without building or installing anything, use . For further information on all of the available switches, refer to the manual page for portupgrade. Ports and Disk Space ports disk-space Using the Ports Collection will use up disk space over time. After building and installing a port, running make clean within the ports skeleton will clean up the temporary work directory. If Portmaster is used to install a port, it will automatically remove this directory unless is specified. If Portupgrade is installed, this command will remove all work directories found within the local copy of the Ports Collection: &prompt.root; portsclean -C In addition, a lot of out-dated source distribution files will collect in /usr/ports/distfiles over time. If Portupgrade is installed, this command will delete all the distfiles that are no longer referenced by any ports: &prompt.root; portsclean -D To use Portupgrade to remove all distfiles not referenced by any port currently installed on the system: &prompt.root; portsclean -DD If Portmaster is installed, use: &prompt.root; portmaster --clean-distfiles By default, this command is interactive and will prompt the user to confirm if a distfile should be deleted. In addition to these commands, the ports-mgmt/pkg_cutleaves package or port automates the task of removing installed ports that are no longer needed. Building Packages with <application>Poudriere</application> Poudriere is a BSD-licensed utility for creating and testing &os; packages. It uses &os; jails to set up isolated compilation environments. These jails can be used to build packages for versions of &os; that are different from the system on which it is installed, and also to build packages for i386 if the host is an &arch.amd64; system. Once the packages are built, they are in a layout identical to the official mirrors. These packages are usable by &man.pkg.8; and other package management tools. Poudriere is installed using the ports-mgmt/poudriere package or port. The installation includes a sample configuration file /usr/local/etc/poudriere.conf.sample. Copy this file to /usr/local/etc/poudriere.conf. Edit the copied file to suit the local configuration. While ZFS is not required on the system running poudriere, it is beneficial. When ZFS is used, ZPOOL must be specified in /usr/local/etc/poudriere.conf and FREEBSD_HOST should be set to a nearby mirror. Defining CCACHE_DIR enables the use of devel/ccache to cache compilation and reduce build times for frequently-compiled code. It may be convenient to put poudriere datasets in an isolated tree mounted at /poudriere. Defaults for the other configuration values are adequate. The number of processor cores detected is used to define how many builds should run in parallel. Supply enough virtual memory, either with RAM or swap space. If virtual memory runs out, compiling jails will stop and be torn down, resulting in weird error messages. Initialize Jails and Port Trees After configuration, initialize poudriere so that it installs a jail with the required &os; tree and a ports tree. Specify a name for the jail using and the &os; version with . On systems running &os;/&arch.amd64;, the architecture can be set with to either i386 or amd64. The default is the architecture shown by uname. &prompt.root; poudriere jail -c -j 10amd64 -v 10.0-RELEASE ====>> Creating 10amd64 fs... done ====>> Fetching base.txz for FreeBSD 10.0-RELEASE amd64 /poudriere/jails/10amd64/fromftp/base.txz 100% of 59 MB 1470 kBps 00m42s ====>> Extracting base.txz... done ====>> Fetching src.txz for FreeBSD 10.0-RELEASE amd64 /poudriere/jails/10amd64/fromftp/src.txz 100% of 107 MB 1476 kBps 01m14s ====>> Extracting src.txz... done ====>> Fetching games.txz for FreeBSD 10.0-RELEASE amd64 /poudriere/jails/10amd64/fromftp/games.txz 100% of 865 kB 734 kBps 00m01s ====>> Extracting games.txz... done ====>> Fetching lib32.txz for FreeBSD 10.0-RELEASE amd64 /poudriere/jails/10amd64/fromftp/lib32.txz 100% of 14 MB 1316 kBps 00m12s ====>> Extracting lib32.txz... done ====>> Cleaning up... done ====>> Jail 10amd64 10.0-RELEASE amd64 is ready to be used &prompt.root; poudriere ports -c -p local ====>> Creating local fs... done ====>> Extracting portstree "local"... Looking up portsnap.FreeBSD.org mirrors... 7 mirrors found. Fetching public key from ec2-eu-west-1.portsnap.freebsd.org... done. Fetching snapshot tag from ec2-eu-west-1.portsnap.freebsd.org... done. Fetching snapshot metadata... done. Fetching snapshot generated at Tue Feb 11 01:07:15 CET 2014: 94a3431f0ce567f6452ffde4fd3d7d3c6e1da143efec76100% of 69 MB 1246 kBps 00m57s Extracting snapshot... done. Verifying snapshot integrity... done. Fetching snapshot tag from ec2-eu-west-1.portsnap.freebsd.org... done. Fetching snapshot metadata... done. Updating from Tue Feb 11 01:07:15 CET 2014 to Tue Feb 11 16:05:20 CET 2014. Fetching 4 metadata patches... done. Applying metadata patches... done. Fetching 0 metadata files... done. Fetching 48 patches. (48/48) 100.00% done. done. Applying patches... done. Fetching 1 new ports or files... done. /poudriere/ports/tester/CHANGES /poudriere/ports/tester/COPYRIGHT [...] Building new INDEX files... done. On a single computer, poudriere can build ports with multiple configurations, in multiple jails, and from different port trees. Custom configurations for these combinations are called sets. See the CUSTOMIZATION section of &man.poudriere.8; for details after ports-mgmt/poudriere or ports-mgmt/poudriere-devel is installed. The basic configuration shown here puts a single jail-, port-, and set-specific make.conf in /usr/local/etc/poudriere.d. The filename in this example is created by combining the jail name, port name, and set name: 10amd64-local-workstation-make.conf. The system make.conf and this new file are combined at build time to create the make.conf used by the build jail. Packages to be built are entered in 10amd64-local-workstation-pkglist: editors/emacs devel/git ports-mgmt/pkg ... Options and dependencies for the specified ports are configured: &prompt.root; poudriere options -j 10amd64 -p local -z workstation -f 10amd64-local-workstation-pkglist Finally, packages are built and a package repository is created: &prompt.root; poudriere bulk -j 10amd64 -p local -z workstation -f 10amd64-local-workstation-pkglist Ctrlt displays the current state of the build. Poudriere also builds files in /poudriere/logs/bulk/jailname that can be used with a web server to display build information. Packages are now available for installation from the poudriere repository. For more information on using poudriere, see &man.poudriere.8; and the main web site, . Configuring pkg Clients to Use a Poudriere Repository While it is possible to use both a custom repository along side of the official repository, sometimes it is useful to disable the official repository. This is done by creating a configuration file that overrides and disables the official configuration file. Create /usr/local/etc/pkg/repos/FreeBSD.conf that contains the following: FreeBSD: { enabled: no } Usually it is easiest to serve a poudriere repository to the client machines via HTTP. Setup a webserver to serve up the package directory, usually something like: /usr/local/poudriere/data/packages/10amd64. Where 10amd64 is the name of the build. If the URL to the package repository is: http://pkg.example.com/10amd64, then the repository configuration file in /usr/local/etc/pkg/repos/custom.conf would look like: custom: { url: "http://pkg.example.com/10amd64", enabled: yes, } Post-Installation Considerations Regardless of whether the software was installed from a binary package or port, most third-party applications require some level of configuration after installation. The following commands and locations can be used to help determine what was installed with the application. Most applications install at least one default configuration file in /usr/local/etc. In the case where an application has a large number of configuration files, a subdirectory will be created to hold them. Often, sample configuration files are installed which end with a suffix such as .sample. The configuration files should be reviewed and possibly edited to meet the system's needs. To edit a sample file, first copy it without the .sample extension. Applications which provide documentation will install it into /usr/local/share/doc and many applications also install manual pages. This documentation should be consulted before continuing. Some applications run services which must be added to /etc/rc.conf before starting the application. These applications usually install a startup script in /usr/local/etc/rc.d. See Starting Services for more information. Users of &man.csh.1; should run rehash to rebuild the known binary list in the shells PATH. Use pkg info to determine which files, man pages, and binaries were installed with the application. Dealing with Broken Ports When a port does not build or install, try the following: Search to see if there is a fix pending for the port in the Problem Report database. If so, implementing the proposed fix may fix the issue. Ask the maintainer of the port for help. Type make maintainer in the ports skeleton or read the port's Makefile to find the maintainer's email address. Remember to include the $FreeBSD: line from the port's Makefile and the output leading up to the error in the email to the maintainer. Some ports are not maintained by an individual but instead by a mailing list. Many, but not all, of these addresses look like freebsd-listname@FreeBSD.org. Take this into account when sending an email. In particular, ports shown as maintained by ports@FreeBSD.org are not maintained by a specific individual. Instead, any fixes and support come from the general community who subscribe to that mailing list. More volunteers are always needed! If there is no response to the email, use Bugzilla to submit a bug report using the instructions in Writing &os; Problem Reports. Fix it! The Porter's Handbook includes detailed information on the ports infrastructure so that you can fix the occasional broken port or even submit your own! Install the package instead of the port using the instructions in . Index: head/en_US.ISO8859-1/books/porters-handbook/testing/chapter.xml =================================================================== --- head/en_US.ISO8859-1/books/porters-handbook/testing/chapter.xml (revision 47723) +++ head/en_US.ISO8859-1/books/porters-handbook/testing/chapter.xml (revision 47724) @@ -1,927 +1,927 @@ Testing the Port Running <command>make describe</command> Several of the &os; port maintenance tools, such as &man.portupgrade.1;, rely on a database called /usr/ports/INDEX which keeps track of such items as port dependencies. INDEX is created by the top-level ports/Makefile via make index, which descends into each port subdirectory and executes make describe there. Thus, if make describe fails in any port, no one can generate INDEX, and many people will quickly become unhappy. It is important to be able to generate this file no matter what options are present in make.conf, so please avoid doing things such as using .error statements when (for instance) a dependency is not satisfied. (See .) If make describe produces a string rather than an error message, everything is probably safe. See bsd.port.mk for the meaning of the string produced. Also note that running a recent version of portlint (as specified in the next section) will cause make describe to be run automatically. Portlint Do check the port with portlint before submitting or committing it. portlint warns about many common errors, both functional and stylistic. For a new (or repocopied) port, portlint -A is the most thorough; for an existing port, portlint -C is sufficient. Since portlint uses heuristics to try to figure out errors, it can produce false positive warnings. In addition, occasionally something that is flagged as a problem really cannot be done in any other way due to limitations in the ports framework. When in doubt, the best thing to do is ask on &a.ports;. Port Tools The ports-mgmt/porttools program is part of the Ports Collection. port is the front-end script, which can help simplify the testing job. Whenever a new port or an update to an existing one needs testing, use port test to test the port, including the portlint checking. This command also detects and lists any files that are not listed in pkg-plist. For example: &prompt.root; port test /usr/ports/net/csup <varname>PREFIX</varname> and <varname>DESTDIR</varname> PREFIX determines where the port will be installed. It defaults to /usr/local, but can be set by the user to a custom path like /opt. The port must respect the value of this variable. DESTDIR, if set by the user, determines the complete alternative environment, usually a jail or an installed system mounted somewhere other than /. A port will actually install into DESTDIR/PREFIX, and register with the package database in DESTDIR/var/db/pkg. As DESTDIR is handled automatically by the ports infrastructure with &man.chroot.8;. There is no need for modifications or any extra care to write DESTDIR-compliant ports. The value of PREFIX will be set to LOCALBASE (defaulting to /usr/local). If USE_LINUX_PREFIX is set, PREFIX will be LINUXBASE (defaulting to /compat/linux). Avoiding hard-coded /usr/local paths in the source makes the port much more flexible and able to cater to the needs of other sites. Often, this can be accomplished by replacing occurrences of /usr/local in the port's various Makefiles with ${PREFIX}. This variable is automatically passed down to every stage of the build and install processes. Make sure the application is not installing things in /usr/local instead of PREFIX. A quick test for such hard-coded paths is: &prompt.user; make clean; make package PREFIX=/var/tmp/`make -V PORTNAME` If anything is installed outside of PREFIX, the package creation process will complain that it cannot find the files. In addition, it is worth checking the same with the stage directory support (see ): &prompt.user; make stage && make check-plist && make stage-qa && make package check-plist checks for files missing from the plist, and files in the plist that are not installed by the port. stage-qa checks for common problems like bad shebang, symlinks pointing outside the stage directory, setuid files, and non-stripped libraries... These tests will not find hard-coded paths inside the port's files, nor will it verify that LOCALBASE is being used to correctly refer to files from other ports. The temporarily-installed port in /var/tmp/`make -V PORTNAME` must be tested for proper operation to make sure there are no problems with paths. PREFIX must not be set explicitly in a port's Makefile. Users installing the port may have set PREFIX to a custom location, and the port must respect that setting. Refer to programs and files from other ports with the variables mentioned above, not explicit pathnames. For instance, if the port requires a macro PAGER to have the full pathname of less, do not use a literal path of /usr/local/bin/less. Instead, use ${LOCALBASE}: -DPAGER=\"${LOCALBASE}/bin/less\" The path with LOCALBASE is more likely to still work if the system administrator has moved the whole /usr/local tree somewhere else. All these tests are done automatically when running poudriere testport or poudriere bulk -t. It is highly recommended that every ports contributor install it, and tests all his ports with it. See for more information. <application>Poudriere</application> For a ports contributor, Poudriere is one of the most important and helpful testing and build tools. Its main features include: Bulk building of the entire ports tree, specific subsets of the ports tree, or a single port including its dependencies Automatic packaging of build results Generation of build log files per port Providing a signed &man.pkg.8; repository Testing of port builds before submitting a patch to the &os; bug tracker or committing to the ports tree Testing for successful ports builds using different options Because Poudriere performs its building in a clean &man.jail.8; environment and uses &man.zfs.8; features, it has several advantages over traditional testing on the host system: No pollution of the host environment: No leftover files, no accidental removals, no changes of existing configuration files. Verify pkg-plist for missing or superfluous entries Ports committers sometimes ask for a Poudriere log alongside a patch submission to assess whether the patch is ready for integration into the ports tree It is also quite straightforward to set up and use, has no dependencies, and will run on any supported &os; release. This section shows how to install, configure, and run Poudriere as part of the normal workflow of a ports contributor. The examples in this section show a default file layout, as standard in &os;. Substitute any local changes accordingly. The ports tree, represented by ${PORTSDIR}, is located in /usr/ports. Both ${LOCALBASE} and ${PREFIX} are /usr/local by default. Installing <application>Poudriere</application> Poudriere is available in the ports tree in ports-mgmt/poudriere. It can be installed using &man.pkg.8; or from ports: &prompt.root; pkg install poudriere or &prompt.root; make -C /usr/ports/ports-mgmt/poudriere install clean There is also a work-in-progress version of Poudriere which will eventually become the next release. It is available in ports-mgmt/poudriere-devel. This development version is used for the official &os; package builds, so it is well tested. It often has newer interesting features. A ports committer will want to use the development version because it is what is used in production, and has all the new features that will make sure everything is exactly right. A contributor will not necessarily need those as the most important fixes are backported to released version. The main reason for the use of the development version to build the official package is because it is faster, in a way that will shorten a full build from 18 hours to 17 hours when using a high end 32 CPU server with 128GB of RAM. Those optimizations will not matter a lot when building ports on a desktop machine. Setting Up <application>Poudriere</application> The port installs a default configuration file, /usr/local/etc/poudriere.conf. Each parameter is documented in the configuration file and in &man.poudriere.8;. Here is a minimal example config file: ZPOOL=tank ZROOTFS=/poudriere BASEFS=/poudriere DISTFILES_CACHE=/usr/ports/distfiles RESOLV_CONF=/etc/resolv.conf FREEBSD_HOST=ftp://ftp.freebsd.org -SVN_HOST=svn0.eu.FreeBSD.org +SVN_HOST=svn.FreeBSD.org ZPOOL The name of the ZFS storage pool which Poudriere shall use. Must be listed in the output of zpool status. ZROOTFS The root of Poudriere-managed file systems. This entry will cause Poudriere to create &man.zfs.8; file systems under tank/poudriere. BASEFS The root mount point for Poudriere file systems. This entry will cause Poudriere to mount tank/poudriere to /poudriere. DISTFILES_CACHE Defines where distfiles are stored. In this example, Poudriere and the host share the distfiles storage directory. This avoids downloading tarballs which are already present on the system. RESOLV_CONF Use the host /etc/resolv.conf inside jails for DNS. This is needed so jails can resolve the URLs of distfiles when downloading. It is not needed when using a proxy. Refer to the default configuration file for proxy configuration. FREEBSD_HOST The FTP/HTTP server to use when the jails are installed from &os; releases and updated with &man.freebsd-update.8;. Choose a server location which is close, for example if the machine is located in Australia, use ftp.au.freebsd.org. SVN_HOST The server from where jails are installed and updated when using Subversion. Also used for ports tree when not using &man.portsnap.8;. Again, choose a nearby location. A list of official Subversion mirrors can be found in the &os; Handbook Subversion section. Creating <application>Poudriere</application> Jails Create the base jails which Poudriere will use for building: &prompt.root; poudriere jail -c -j 93Ramd64 -v 9.3-RELEASE -a amd64 Fetch a 9.3-RELEASE for amd64 from the FTP server given by FREEBSD_HOST in poudriere.conf, create the zfs file system tank/poudriere/jails/93Ramd64, mount it on /poudriere/jails/93Ramd64 and extract the 9.3-RELEASE tarballs into this file system. &prompt.root; poudriere jail -c -j 10i386 -v stable/10 -a i386 -m svn+https Create tank/poudriere/jails/10i386, mount it on /poudriere/jails/10i386, then check out the tip of the Subversion branch of &os;-10-STABLE from SVN_HOST in poudriere.conf into /poudriere/jails/10i386/usr/src, then complete a buildworld and install it into /poudriere/jails/10i386. If a specific Subversion revision is needed, append it to the version string. For example: &prompt.root; poudriere jail -c -j 10i386 -v stable/10@123456 -a i386 -m svn+https While it is possible to build a newer version of &os; on an older version, most of the time it will not run. For example, if a stable/10 jail is needed, the host will have to run stable/10 too. Running 10.0-RELEASE is not enough. The default svn protocol works but is not very secure. Using svn+https along with verifying the remote server's SSL fingerprint is advised. It will ensure that the files used for building the jail are from a trusted source. A list of jails currently known to Poudriere can be shown with poudriere jail -l: &prompt.root; poudriere jail -l JAILNAME VERSION ARCH METHOD 93Ramd64 9.3-RELEASE amd64 ftp 10i386 10.0-STABLE i386 svn+https Keeping <application>Poudriere</application> Jails Updated Managing updates is very straightforward. The command: &prompt.root; poudriere jail -u -j JAILNAME updates the specified jail to the latest version available. For &os; releases, update to the latest patchlevel with &man.freebsd-update.8;. For &os; versions built from source, update to the latest Subversion revision in the branch. For jails employing a svn+* method, it is helpful to add -J NumberOfParallelBuildJobs to speed up the build by increasing the number of parallel compile jobs used. For example, if the building machine has 6 CPUs, use: &prompt.root; poudriere jail -u -J 6 -j JAILNAME Setting Up Ports Trees for Use with <application>Poudriere</application> There are multiple ways to use ports trees in Poudriere. The most straightforward way is to have Poudriere create a default ports tree for itself: &prompt.root; poudriere ports -c This command creates tank/poudriere/ports/default, mount it on /poudriere/ports/default, and populate it using &man.portsnap.8;. Afterward it is included in the list of known ports trees: &prompt.root; poudriere ports -l PORTSTREE METHOD PATH default portsnap /poudriere/ports/default Note that the default ports tree is special. Each of the build commands explained later will implicitly use this ports tree unless specifically specified otherwise. To use another tree, add -p treename to the commands. While useful for regular bulk builds, having this default ports tree with the &man.portsnap.8; method may not be the best way to deal with local modifications for a ports contributor. As with the creation of jails, it is possible to use a different method for creating the ports tree. To add an additional ports tree for testing local modifications and ports development, checking out the tree via Subversion is possible: &prompt.root; poudriere ports -c -m svn+https -p subversive Creates tank/poudriere/ports/subversive and mounts it on /poudriere/ports/subversive. It is then populated using Subversion. Finally, it is added to the list of known ports trees: &prompt.root; poudriere ports -l PORTSTREE METHOD PATH default portsnap /poudriere/ports/default subversive svn+https /poudriere/ports/subversive The svn method allows extra qualifiers to tell Subversion exactly how to fetch data. This is explained in &man.poudriere.8;. For instance, poudriere ports -c -m svn+ssh -p subversive uses SSH for the checkout. Using Manually Managed Ports Trees with Poudriere Depending on the workflow, it can be extremely helpful to use ports trees which are maintained manually. For instance, if there is a local copy of the ports tree in /work/ports, point Poudriere to the location: &prompt.root; poudriere ports -c -F -f none -M /work/ports -p development This will be listed in the table of known trees: &prompt.root; poudriere ports -l PORTSTREE METHOD PATH development - /work/ports The dash in the METHOD column means that Poudriere will not update or change this ports tree, ever. It is completely up to the user to maintain this tree, including all local modifications that may be used for testing new ports and submitting patches. Keeping Poudriere Ports Trees Updated As straightforward as with jails described earlier: &prompt.root; poudriere ports -u -p PORTSTREE Will update the given PORTSTREE, one tree given by the output of poudriere -l, to the latest revision available on the official servers. Ports trees without a method, see , cannot be updated like this. They must be updated manually by the porter. Testing Ports After jails and ports trees have been set up, the result of a contributor's modifications to the ports tree can be tested. For example, local modifications to the www/firefox port located in /work/ports/www/firefox can be tested in the previously created 9.3-RELEASE jail: &prompt.root; poudriere testport -j 93Ramd64 -p development -o www/firefox This will build all dependencies of Firefox. If a dependency has been built previously and is still up-to-date, the pre-built package is installed. If a dependency has no up-to-date package, one will be built with default options in a jail. Then Firefox itself is built. The complete build of every port is logged to /poudriere/data/logs/bulk/93Ri386-development/build-time/logs. The directory name 93Ri386-development is derived from the arguments to -j and -p, respectively. For convenience, a symbolic link /poudriere/data/logs/bulk/93Ri386-development/latest is also maintained. The link points to the latest build-time directory. Also in this directory is an index.html for observing the build process with a web browser. By default, Poudriere cleans up the jails and leaves log files in the directories mentioned above. To ease investigation, jails can be kept running after the build by adding to testport: &prompt.root; poudriere testport -j 93Ramd64 -p development -i -o www/firefox After the build completes, and regardless of whether it was successful, a shell is provided within the jail. The shell is used to investigate further. Poudriere can be told to leave the jail running after the build finishes with . Poudriere will show the command to run when the jail is no longer needed. It is then possible to &man.jexec.8; into it: &prompt.root; poudriere testport -j 93Ramd64 -p development -I -o www/firefox [...] ====>> Installing local Pkg repository to /usr/local/etc/pkg/repos ====>> Leaving jail 93Ramd64-development-n running, mounted at /poudriere/data/.m/93Ramd64-development/ref for interactive run testing ====>> To enter jail: jexec 93Ramd64-development-n env -i TERM=$TERM /usr/bin/login -fp root ====>> To stop jail: poudriere jail -k -j 93Ramd64 -p development &prompt.root; jexec 93Ramd64-development-n env -i TERM=$TERM /usr/bin/login -fp root &prompt.root; [do some stuff in the jail] &prompt.root; exit &prompt.root; poudriere jail -k -j 93Ramd64 -p development ====>> Umounting file systems An integral part of the &os; ports build infrastructure is the ability to tweak ports to personal preferences with options. These can be tested with Poudriere as well. Adding the : &prompt.root; poudriere testport -c -o www/firefox Presents the port configuration dialog before the port is built. The ports given after in the format category/portname will use the specified options, all dependencies will use the default options. Testing dependent ports with non-default options can be accomplished using sets, see . When testing ports where pkg-plist is altered during build depending on the selected options, it is recommended to perform a test run with all options selected and one with all options deselected. Using Sets For all actions involving builds, a so-called set can be specified using -z setname. A set refers to a fully independent build. This allows, for instance, usage of testport with non-standard options for the dependent ports. To use sets, Poudriere expects an existing directory structure similar to PORT_DBDIR, defaults to /var/db/ports in its configuration directory. This directory is then nullfs-mounted into the jails where the ports and their dependencies are built. Usually a suitable starting point can be obtained by recursively copying the existing PORT_DBDIR to /usr/local/etc/poudriere.d/jailname-portname-setname-options. This is described in detail in &man.poudriere.8;. For instance, testing www/firefox in a specific set named devset, add the -z devset parameter to the testport command: &prompt.root; poudriere testport -j 93Ramd64 -p development -z devset -o www/firefox This will look for the existence of these directories in this order: /usr/local/etc/poudriere.d/93Ramd64-development-devset-options /usr/local/etc/poudriere.d/93Ramd64-devset-options /usr/local/etc/poudriere.d/93Ramd64-development-options /usr/local/etc/poudriere.d/devset-options /usr/local/etc/poudriere.d/development-options /usr/local/etc/poudriere.d/93Ramd64-options /usr/local/etc/poudriere.d/options From this list, Poudriere nullfs-mounts the first existing directory tree into the /var/db/ports directory of the build jails. Hence, all custom options are used for all the ports during this run of testport. After the directory structure for a set is provided, the options for a particular port can be altered. For example: &prompt.root; poudriere options -c www/firefox -z devset The configuration dialog for www/firefox is shown, and options can be edited. The selected options are saved to the devset set. Poudriere is very flexible in the option configuration. They can be set for particular jails, ports trees, and for multiple ports by one command. Refer to &man.poudriere.8; for details. Providing a Custom <filename>make.conf</filename> File Similar to using sets, Poudriere will also use a custom make.conf if it is provided. No special command line argument is necessary. Instead, Poudriere looks for existing files matching a name scheme derived from the command line. For instance: &prompt.root; poudriere testport -j 93Ramd64 -p development -z devset -o www/firefox causes Poudriere to check for the existence of these files in this order: /usr/local/etc/poudriere.d/make.conf /usr/local/etc/poudriere.d/devset-make.conf /usr/local/etc/poudriere.d/development-make.conf /usr/local/etc/poudriere.d/93Ramd64-make.conf /usr/local/etc/poudriere.d/93Ramd64-development-make.conf /usr/local/etc/poudriere.d/93Ramd64-devset-make.conf /usr/local/etc/poudriere.d/93Ramd64-development-devset-make.conf Unlike with sets, all of the found files will be appended, in that order, into one make.conf inside the build jails. It is hence possible to have general make variables, intended to affect all builds in /usr/local/etc/poudriere.d/make.conf. Special variables, intended to affect only certain jails or sets can be set in specialised make.conf files, such as /usr/local/etc/poudriere.d/93Ramd64-development-devset-make.conf. Using <filename>make.conf</filename> to Change Default <application>Perl</application> To build a set with a non default Perl version, for example, 5.20, using a set named perl5-20, create a perl5-20-make.conf with this line: DEFAULT_VERSIONS+= perl=5.20 Note the use of += so that if the variable is already set in the default make.conf its content will not be overwritten. Pruning no Longer Needed Distfiles Poudriere comes with a built-in mechanism to remove outdated distfiles that are no longer used by any port of a given tree. The command &prompt.root; poudriere distclean -p portstree will scan the distfiles folder, DISTFILES_CACHE in poudriere.conf, versus the ports tree given by the -p portstree argument and prompt for removal of those distfiles. To skip the prompt and remove all unused files unconditionally, the -y argument can be added: &prompt.root; poudriere distclean -p portstree -y Tinderbox As an avid ports contributor, take a look at Tinderbox. It is a powerful system for building and testing ports. Install Tinderbox using ports-mgmt/tinderbox port. Be sure to read supplied documentation since the configuration is not trivial. Visit the Tinderbox website for more details.