Index: en_US.ISO8859-1/books/porters-handbook/testing/chapter.xml
===================================================================
--- en_US.ISO8859-1/books/porters-handbook/testing/chapter.xml
+++ en_US.ISO8859-1/books/porters-handbook/testing/chapter.xml
@@ -187,6 +187,712 @@
+
+ <application>Poudriere</application>
+
+ For a ports contributor, Poudriere
+ is one of the most important and helpful testing and build
+ tools. Its main features include:
+
+
+
+ Bulk building of the entire ports tree, specific subsets
+ of the ports tree, or a single port including its
+ dependencies
+
+
+
+ Automatic packaging of build results
+
+
+
+ Generation of build log files per port
+
+
+
+ Providing a signed &man.pkg.8; repository
+
+
+
+ Testing of port builds before submitting a patch to the
+ &os; bug tracker or committing to the ports tree
+
+
+
+ Testing for successful ports builds using different
+ options
+
+
+
+ Because Poudriere performs its
+ building in a clean &man.jail.8; environment and uses
+ &man.zfs.8; features, it has several advantages over traditional
+ testing on the host system:
+
+
+
+ No pollution of the host environment: No leftover files,
+ no accidental removals, no changes of existing configuration
+ files.
+
+
+
+ Verify pkg-plist for missing or
+ superfluous entries
+
+
+
+ Ports committers sometimes ask for a
+ Poudriere log alongside a patch
+ submission to assess whether the patch is ready for
+ integration into the ports tree
+
+
+
+ It is also quite straightforward to set up and use, has no
+ dependencies, and will run on any supported &os; release.
+ This section shows how to install, configure, and run
+ Poudriere as part of the normal
+ workflow of a ports contributor.
+
+ The examples in this section show a default file layout, as
+ standard in &os;. Substitute any local changes
+ accordingly. The ports tree, represented by
+ ${PORTSDIR}, is located in
+ /usr/ports. Both
+ ${LOCALBASE} and ${PREFIX}
+ are /usr/local by default.
+
+
+ Installing <application>Poudriere</application>
+
+ Poudriere is available in the
+ ports tree in ports-mgmt/poudriere. It can be
+ installed using &man.pkg.8; or from ports:
+
+
+ &prompt.root; pkg install poudriere
+
+ or
+
+ &prompt.root; make -C /usr/ports/ports-mgmt/poudriere install clean
+
+ There is also a work-in-progress version of
+ Poudriere which will eventually
+ become the next release. It is available in ports-mgmt/poudriere-devel. This
+ development version is used for the official &os; package
+ builds, so it is well tested. It often has newer interesting
+ features. A ports committer will want to use the development
+ version because it is what is used in production, and has all
+ the new features that will make sure everything is exactly
+ right. A contributor will not necessarily need those as the
+ most important fixes are backported to released version. The
+ main reason for the use of the development version to build
+ the official package is because it is faster, in a way that
+ will shorten a full build from 18 hours to 17 hours when using
+ a high end 32 CPU server with 128GB of
+ RAM. Those optimizations will not matter a
+ lot when building ports on a desktop machine.
+
+
+
+
+ Setting Up <application>Poudriere</application>
+
+ The port installs a default configuration file,
+ /usr/local/etc/poudriere.conf. Each
+ parameter is documented in the configuration file and in
+ &man.poudriere.8;. Here is a minimal example config
+ file:
+
+ ZPOOL=tank
+ZROOTFS=/poudriere
+BASEFS=/poudriere
+DISTFILES_CACHE=/usr/ports/distfiles
+RESOLV_CONF=/etc/resolv.conf
+FREEBSD_HOST=ftp://ftp.freebsd.org
+SVN_HOST=svn0.eu.FreeBSD.org
+
+
+
+ ZPOOL
+
+
+ The name of the ZFS storage pool
+ which Poudriere shall use.
+ Must be listed in the output of
+ zpool status.
+
+
+
+
+ ZROOTFS
+
+
+ The root of Poudriere-managed file systems. This
+ entry will cause Poudriere to
+ create &man.zfs.8; file systems under
+ tank/poudriere.
+
+
+
+
+ BASEFS
+
+
+ The root mount point for
+ Poudriere file systems. This
+ entry will cause Poudriere to mount
+ tank/poudriere to
+ /poudriere.
+
+
+
+
+ DISTFILES_CACHE
+
+
+ Defines where distfiles are stored. In
+ this example, Poudriere and the host share the distfiles
+ storage directory. This avoids downloading tarballs which
+ are already present on the system.
+
+
+
+
+ RESOLV_CONF
+
+
+ Use the host /etc/resolv.conf
+ inside jails for DNS. This is needed
+ so jails can resolve the URLs of
+ distfiles when downloading. It is not needed when using a
+ proxy. Refer to the default configuration file for
+ proxy configuration.
+
+
+
+
+ FREEBSD_HOST
+
+
+ The FTP/HTTP
+ server to use when the jails are installed from &os; releases and
+ updated with &man.freebsd-update.8;. Choose a server
+ location which is close, for example if the machine is
+ located in Australia, use
+ ftp.au.freebsd.org.
+
+
+
+
+ SVN_HOST
+
+
+ The server from where jails are installed and
+ updated when using
+ Subversion. Also used for ports tree when not using &man.portsnap.8;. Again, choose a
+ nearby location. A list of official Subversion
+ mirrors can be found in the &os;
+ Handbook Subversion
+ section.
+
+
+
+
+
+
+
+ Creating <application>Poudriere</application> Jails
+
+ Create the base jails which Poudriere will use for
+ building:
+
+ &prompt.root; poudriere jail -c -j 93Ramd64 -v 9.3-RELEASE -a amd64
+
+ Fetch a 9.3-RELEASE for
+ amd64 from the FTP server given by
+ FREEBSD_HOST in
+ poudriere.conf, create the zfs file
+ system tank/poudriere/jails/93Ramd64, mount
+ it on /poudriere/jails/93Ramd64 and
+ extract the 9.3-RELEASE tarballs into this
+ file system.
+
+ &prompt.root; poudriere jail -c -j 10i386 -v stable/10 -a i386 -m svn+https
+
+ Create
+ tank/poudriere/jails/10i386, mount it on
+ /poudriere/jails/10i386, then check out
+ the tip of the Subversion branch of
+ &os;-10-STABLE from
+ SVN_HOST in
+ poudriere.conf into
+ /poudriere/jails/10i386/usr/src, then
+ complete a buildworld and install
+ it into /poudriere/jails/10i386.
+
+
+ If a specific Subversion revision is needed, append it to the
+ version string. For example:
+
+ &prompt.root; poudriere jail -c -j 10i386 -v stable/10@123456 -a i386 -m svn+https
+
+
+
+ While it is possible to build a newer version of &os; on
+ an older version, most of the time it will not run. For
+ example, if a stable/10 jail is needed,
+ the host will have to run stable/10 too.
+ Running 10.0-RELEASE is not
+ enough.
+
+
+
+ The default svn protocol works but is
+ not very secure. Using svn+https along
+ with verifying the remote server's SSL
+ fingerprint is advised. It will
+ ensure that the files used for building the jail are from a
+ trusted source.
+
+
+ A list of jails currently known to
+ Poudriere can be shown with
+ poudriere jail -l:
+
+ &prompt.root; poudriere jail -l
+ JAILNAME VERSION ARCH METHOD
+ 93Ramd64 9.3-RELEASE amd64 ftp
+ 10i386 10.0-STABLE i386 svn+https
+
+
+
+
+ Keeping <application>Poudriere</application> Jails Updated
+
+ Managing updates is very straightforward. The
+ command:
+
+ &prompt.root; poudriere jail -u -j JAILNAME
+
+ updates
+ the specified jail to the latest version available. For &os;
+ releases, update to the latest patchlevel with
+ &man.freebsd-update.8;. For &os; versions built from source,
+ update to the latest Subversion
+ revision in the branch.
+
+
+ For jails employing a
+ svn+* method,
+ it is helpful to add -J
+ NumberOfParallelBuildJobs
+ to speed up the build by increasing the number of parallel
+ compile jobs used. For example, if the building machine has
+ 6 CPUs, use:
+
+ &prompt.root; poudriere jail -u -J 6 -j JAILNAME
+
+
+
+
+ Setting Up Ports Trees for Use with <application>Poudriere</application>
+
+ There are multiple ways to use ports trees in Poudriere.
+ The most straightforward way is to have Poudriere create a
+ default ports tree for itself:
+
+ &prompt.root; poudriere ports -c
+
+ This command creates tank/poudriere/ports/default,
+ mount it on /poudriere/ports/default, and populate it using
+ &man.portsnap.8;. Afterward it is included in the list of known
+ ports trees:
+
+ &prompt.root; poudriere ports -l
+ PORTSTREE METHOD PATH
+ default portsnap /poudriere/ports/default
+
+
+ Note that the default ports tree is special. Each of
+ the build commands explained later will implicitly use this
+ ports tree unless specifically specified otherwise. To use
+ another tree, add -p
+ treename to the
+ commands.
+
+
+ While useful for regular bulk builds,
+ having this default ports tree with the &man.portsnap.8;
+ method may not be the best way to deal with local
+ modifications for a ports contributor. As with the creation of jails, it is
+ possible to use a different method for creating the ports
+ tree. To add an additional ports tree for testing local
+ modifications and ports development, checking out the tree via
+ Subversion is possible:
+
+ &prompt.root; poudriere ports -c -m svn+https -p subversive
+
+ Creates
+ tank/poudriere/ports/subversive and mounts it
+ on /poudriere/ports/subversive. It is then
+ populated using Subversion.
+ Finally, it is added to the list of known ports
+ trees:
+
+ &prompt.root; poudriere ports -l
+ PORTSTREE METHOD PATH
+ default portsnap /poudriere/ports/default
+ subversive svn+https /poudriere/ports/subversive
+
+
+ The svn method allows extra
+ qualifiers to tell Subversion
+ exactly how to fetch data. This is explained in
+ &man.poudriere.8;. For instance, poudriere ports
+ -c -m svn+ssh -p subversive uses
+ SSH for the checkout.
+
+
+
+
+
+ Using Manually Managed Ports Trees with Poudriere
+
+ Depending on the workflow, it can be extremely helpful to
+ use ports trees which are maintained manually. For instance,
+ if there is a local copy of the ports tree in
+ /work/ports, point Poudriere to the
+ location:
+
+ &prompt.root; poudriere ports -c -F -f none -M /work/ports -p development
+
+ This will be listed in the table of known trees:
+
+ &prompt.root; poudriere ports -l
+ PORTSTREE METHOD PATH
+ development - /work/ports
+
+
+ The dash in the METHOD column means
+ that Poudriere will not update or change this ports tree,
+ ever. It is completely up to the user to maintain this
+ tree, including all local modifications that may be used for
+ testing new ports and submitting patches.
+
+
+
+
+
+ Keeping Poudriere Ports Trees Updated
+
+ As straightforward as with jails described earlier:
+
+ &prompt.root; poudriere ports -u -p PORTSTREE
+
+ Will update the given
+ PORTSTREE, one tree given by the
+ output of poudriere -l, to the latest
+ revision available on the official servers.
+
+
+ Ports trees without a method, see , cannot be
+ updated like this. They must be updated manually by the
+ porter.
+
+
+
+
+
+ Testing Ports
+
+ After jails and ports trees have been set up, the result
+ of a contributor's modifications to the ports tree can be
+ tested.
+
+ For example, local modifications to the www/firefox port located in
+ /work/ports/www/firefox can be tested in
+ the previously created 9.3-RELEASE
+ jail:
+
+ &prompt.root; poudriere testport -j 93Ramd64 -p development -o www/firefox
+
+ This will build all dependencies of Firefox. If a dependency has
+ been built previously and is still up-to-date, the
+ pre-built package is installed. If a dependency has no
+ up-to-date package, one will be built with default options in a jail. Then
+ Firefox itself is built.
+
+ The complete build of every port is logged
+ to
+ /poudriere/data/logs/bulk/93Ri386-development/build-time/logs.
+
+ The directory name 93Ri386-development
+ is derived from the arguments to -j and
+ -p, respectively. For convenience, a symbolic link
+ /poudriere/data/logs/bulk/93Ri386-development/latest
+ is also maintained. The link points to the latest
+ build-time
+ directory. Also in this directory is an index.html for
+ observing the build process with a web browser.
+
+ By default, Poudriere
+ cleans up the jails and leaves log files in the
+ directories mentioned above. To ease investigation, jails can
+ be kept running after
+ the build by adding to
+ testport:
+
+ &prompt.root; poudriere testport -j 93Ramd64 -p development -i -o www/firefox
+
+ After the build completes, and regardless of whether it
+ was successful, a shell is provided within the jail.
+ The shell is used to investigate further.
+ Poudriere can be told to leave the
+ jail running after the build finishes with
+ . Poudriere will show the command to
+ run when the jail is no longer needed. It is then
+ possible to &man.jexec.8; into it:
+
+ &prompt.root; poudriere testport -j 93Ramd64 -p development -I -o www/firefox
+[...]
+====>> Installing local Pkg repository to /usr/local/etc/pkg/repos
+====>> Leaving jail 93Ramd64-development-n running, mounted at /poudriere/data/.m/93Ramd64-development/ref for interactive run testing
+====>> To enter jail: jexec 93Ramd64-development-n env -i TERM=$TERM /usr/bin/login -fp root
+====>> To stop jail: poudriere jail -k -j 93Ramd64 -p development
+&prompt.root; jexec 93Ramd64-development-n env -i TERM=$TERM /usr/bin/login -fp root
+&prompt.root; [do some stuff in the jail]
+&prompt.root; exit
+&prompt.root; poudriere jail -k -j 93Ramd64 -p development
+====>> Umounting file systems
+
+ An integral part of the &os; ports build infrastructure is
+ the ability to tweak ports to personal preferences with
+ options. These can be tested with Poudriere as well.
+ Adding the :
+
+ &prompt.root; poudriere testport -c -o www/firefox
+
+ Presents the port configuration dialog before the
+ port is built. The ports given after in the format
+ category/portname
+ will use the specified options, all
+ dependencies will use the default options. Testing dependent
+ ports with non-default options can be accomplished using sets,
+ see .
+
+
+ When testing ports where pkg-plist
+ is altered during build depending on the selected options,
+ it is recommended to perform a test run with all options
+ selected and one with all options
+ deselected.
+
+
+
+
+ Using Sets
+
+ For all actions involving builds, a so-called set can be
+ specified using -z
+ setname. A set refers
+ to a fully independent build. This allows, for instance,
+ usage of testport with non-standard options
+ for the dependent ports.
+
+ To use sets, Poudriere expects an existing directory
+ structure similar to PORT_DBDIR, defaults to
+ /var/db/ports in its configuration
+ directory. This directory is then nullfs-mounted into the
+ jails where the ports and their dependencies are built.
+ Usually a suitable starting point can be obtained by
+ recursively copying the existing PORT_DBDIR
+ to
+ /usr/local/etc/poudriere.d/jailname-portname-setname-options.
+ This is described in detail in &man.poudriere.8;. For
+ instance, testing www/firefox
+ in a specific set named devset, add the
+ -z devset parameter to the testport
+ command:
+
+ &prompt.root; poudriere testport -j 93Ramd64 -p development -z devset -o www/firefox
+
+ This will look for the existence of these directories in
+ this order:
+
+
+
+ /usr/local/etc/poudriere.d/93Ramd64-development-devset-options
+
+
+
+ /usr/local/etc/poudriere.d/93Ramd64-devset-options
+
+
+
+ /usr/local/etc/poudriere.d/93Ramd64-development-options
+
+
+
+ /usr/local/etc/poudriere.d/devset-options
+
+
+
+ /usr/local/etc/poudriere.d/development-options
+
+
+
+ /usr/local/etc/poudriere.d/93Ramd64-options
+
+
+
+ /usr/local/etc/poudriere.d/options
+
+
+
+ From this list, Poudriere nullfs-mounts the
+ first existing directory tree into the
+ /var/db/ports directory of the build
+ jails. Hence, all custom options are used for all the
+ ports during this run of testport.
+
+ After the directory structure for a set is provided, the
+ options for a particular port can be altered. For
+ example:
+
+ &prompt.root; poudriere options -c www/firefox -z devset
+
+ The configuration dialog for www/firefox is shown, and options can be
+ edited. The selected options are saved to the
+ devset set.
+
+
+ Poudriere is very flexible in the option
+ configuration. They can be set for particular jails, ports trees,
+ and for multiple ports by one command. Refer to
+ &man.poudriere.8; for details.
+
+
+
+
+
+ Providing a Custom <filename>make.conf</filename>
+ File
+
+ Similar to using sets, Poudriere will also use a custom
+ make.conf if it is provided. No special
+ command line argument is necessary. Instead, Poudriere
+ looks for existing files matching a name scheme derived from
+ the command line. For instance:
+
+ &prompt.root; poudriere testport -j 93Ramd64 -p development -z devset -o www/firefox
+
+ causes Poudriere to check for the existence of these
+ files in this order:
+
+
+
+ /usr/local/etc/poudriere.d/make.conf
+
+
+
+ /usr/local/etc/poudriere.d/devset-make.conf
+
+
+
+ /usr/local/etc/poudriere.d/development-make.conf
+
+
+
+ /usr/local/etc/poudriere.d/93Ramd64-make.conf
+
+
+
+ /usr/local/etc/poudriere.d/93Ramd64-development-make.conf
+
+
+
+ /usr/local/etc/poudriere.d/93Ramd64-devset-make.conf
+
+
+
+ /usr/local/etc/poudriere.d/93Ramd64-development-devset-make.conf
+
+
+
+ Unlike with sets, all of the found files will be appended,
+ in that order, into one
+ make.conf inside the build jails. It is
+ hence possible to have general make variables, intended to
+ affect all builds in
+ /usr/local/etc/poudriere.d/make.conf.
+ Special variables, intended to affect only certain jails or
+ sets can be set in specialised make.conf
+ files, such as
+ /usr/local/etc/poudriere.d/93Ramd64-development-devset-make.conf.
+
+
+ Using <filename>make.conf</filename> to Change Default
+ <application>Perl</application>
+
+ To build a set with a non default
+ Perl version, for example,
+ 5.20, using a set named
+ perl5-20, create a
+ perl5-20-make.conf with this
+ line:
+
+ DEFAULT_VERSIONS+= perl=5.20
+
+
+ Note the use of += so that if the
+ variable is already set in the default
+ make.conf its content will not be
+ overwritten.
+
+
+
+
+
+
+ Pruning no Longer Needed Distfiles
+
+ Poudriere comes with a built-in mechanism to
+ remove outdated distfiles that are no longer used by any port
+ of a given tree. The command
+
+ &prompt.root; poudriere distclean -p portstree
+
+ will scan the distfiles folder,
+ DISTFILES_CACHE in
+ poudriere.conf, versus the ports tree
+ given by the -p
+ portstree argument and
+ prompt for removal of those distfiles. To skip the prompt and
+ remove all unused files unconditionally, the
+ -y argument can be added:
+
+ &prompt.root; poudriere distclean -p portstree -y
+
+
+
+
+
+
Tinderbox
@@ -202,23 +908,4 @@
Tinderbox
website for more details.
-
-
- Poudriere
-
- As a ports contributor, consider installing
- poudriere. It is a powerful
- system for building and testing ports.
- Poudriere can be installed with
- ports-mgmt/poudriere.
-
- There is also a ports-mgmt/poudriere-devel that often
- has newer features that are mostly helpful when testing
- ports.
-
- Visit the Poudriere
- website for more details.
-