diff --git a/documentation/content/en/books/porters-handbook/testing/_index.adoc b/documentation/content/en/books/porters-handbook/testing/_index.adoc index 6f326f6ac8..b4e88cfe4c 100644 --- a/documentation/content/en/books/porters-handbook/testing/_index.adoc +++ b/documentation/content/en/books/porters-handbook/testing/_index.adoc @@ -1,630 +1,666 @@ --- title: Chapter 10. Testing the Port prev: books/porters-handbook/pkg-files next: books/porters-handbook/upgrading description: Testing a FreeBSD Port tags: ["testing", "port", "Portclippy", "Portfmt", "Portlint", "poudriere", "sets"] showBookMenu: true weight: 10 path: "/books/porters-handbook/" --- [[testing]] = Testing the Port :doctype: book :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :sectnumoffset: 10 :partnums: :source-highlighter: rouge :experimental: :images-path: books/porters-handbook/ ifdef::env-beastie[] ifdef::backend-html5[] :imagesdir: ../../../../images/{images-path} endif::[] ifndef::book[] include::shared/authors.adoc[] include::shared/mirrors.adoc[] include::shared/releases.adoc[] include::shared/attributes/attributes-{{% lang %}}.adoc[] include::shared/{{% lang %}}/teams.adoc[] include::shared/{{% lang %}}/mailing-lists.adoc[] include::shared/{{% lang %}}/urls.adoc[] toc::[] endif::[] ifdef::backend-pdf,backend-epub3[] include::../../../../../shared/asciidoctor.adoc[] endif::[] endif::[] ifndef::env-beastie[] toc::[] include::../../../../../shared/asciidoctor.adoc[] endif::[] [[make-describe]] == Running `make describe` Several of the FreeBSD port maintenance tools, such as man:portupgrade[1], rely on a database called [.filename]#/usr/ports/INDEX# which keeps track of such items as port dependencies. [.filename]#INDEX# is created by the top-level [.filename]#ports/Makefile# via `make index`, which descends into each port subdirectory and executes `make describe` there. Thus, if `make describe` fails in any port, no one can generate [.filename]#INDEX#, and many people will quickly become unhappy. [NOTE] ==== It is important to be able to generate this file no matter what options are present in [.filename]#make.conf#, so please avoid doing things such as using `.error` statements when (for instance) a dependency is not satisfied. (See crossref:porting-dads[dads-dot-error,Avoid Use of the `.error` Construct].) ==== If `make describe` produces a string rather than an error message, everything is probably safe. See [.filename]#bsd.port.mk# for the meaning of the string produced. Also note that running a recent version of `portlint` (as specified in the next section) will cause `make describe` to be run automatically. [[make-test]] == Running `make test` Even if the port builds fine, it is a good idea to ensure that the software correctly does what it is supposed to do. If the original upstream project provides tests along with the software, it is a good idea to run them and check everything works as expected. A port can enable tests automatically by using the `TEST_TARGET` variable. When set, this variable contains the name of the testing target of the port. This is usually just `test` but other names include `tests`, `check` or for specific cases things like `run_tests.py`. In addition to the `TEST_TARGET` variable the framework provides the following variables to control the tests execution: * `TEST_WRKSRC` is the directory to do the tests in. * `TEST_ENV` contains additional variables to be passed to the test stage. * `TEST_ARGS` contains any extra arguments passed to the test stage. Examples of use of these variables can be found in package:cad/xyce[], package:www/libjwt[] and others. [NOTE] ==== Please make sure that tests do not break when updating a port. ==== [[testing-portclippy]] == Portclippy / Portfmt Those tools come from package:ports-mgmt/portfmt[]. Portclippy is a linter that checks if variables in the [.filename]#Makefile# are in the correct order according to crossref:order[porting-order,Order of Variables in Port Makefiles]. Portfmt is a tool for automatically formatting [.filename]#Makefile#. [[testing-portlint]] == Portlint Do check the port with crossref:quick-porting[porting-portlint,`portlint`] before submitting or committing it. `portlint` warns about many common errors, both functional and stylistic. For a new port, `portlint -A` is the most thorough; for an existing port, `portlint -C` is sufficient. Since `portlint` uses heuristics to try to figure out errors, it can produce false positive warnings. In addition, occasionally something that is flagged as a problem really cannot be done in any other way due to limitations in the ports framework. pass:[] When in doubt, the best thing to do is ask on {freebsd-ports}. pass:[] [[testing-porttools]] == Port Tools The package:ports-mgmt/porttools[] program is part of the Ports Collection. `port` is the front-end script, which can help simplify the testing job. Whenever a new port or an update to an existing one needs testing, use `port test` to test the port, including the <> checking. This command also detects and lists any files that are not listed in [.filename]#pkg-plist#. For example: [source,shell] .... # port test /usr/ports/net/csup .... [[porting-prefix]] == `PREFIX` and `DESTDIR` `PREFIX` determines where the port will be installed. It defaults to [.filename]#/usr/local#, but can be set by the user to a custom path like [.filename]#/opt#. The port must respect the value of this variable. `DESTDIR`, if set by the user, determines the complete alternative environment, usually a jail or an installed system mounted somewhere other than [.filename]#/#. A port will actually install into [.filename]#DESTDIR/PREFIX#, and register with the package database in [.filename]#DESTDIR/var/db/pkg#. `DESTDIR` is handled automatically by the ports infrastructure with man:chroot[8]. There is no need for modifications or any extra care to write `DESTDIR`-compliant ports. The value of `PREFIX` will be set to `LOCALBASE` (defaulting to [.filename]#/usr/local#). If `USE_LINUX_PREFIX` is set, `PREFIX` will be `LINUXBASE` (defaulting to [.filename]#/compat/linux#). Avoiding hard-coded [.filename]#/usr/local# paths in the source makes the port much more flexible and able to cater to the needs of other sites. Often, this can be accomplished by replacing occurrences of [.filename]#/usr/local# in the port's various [.filename]##Makefile##s with `${PREFIX}`. This variable is automatically passed down to every stage of the build and install processes. Make sure the application is not installing things in [.filename]#/usr/local# instead of `PREFIX`. A quick test for such hard-coded paths is: [source,shell] .... % make clean; make package PREFIX=/var/tmp/`make -V PORTNAME` .... If anything is installed outside of `PREFIX`, the package creation process will complain that it cannot find the files. In addition, it is worth checking the same with the stage directory support (see crossref:special[staging,Staging]): [source,shell] .... % make stage && make check-plist && make stage-qa && make package .... * `check-plist` checks for files missing from the plist, and files in the plist that are not installed by the port. * `stage-qa` checks for common problems like bad shebang, symlinks pointing outside the stage directory, setuid files, and non-stripped libraries... These tests will not find hard-coded paths inside the port's files, nor will it verify that `LOCALBASE` is being used to correctly refer to files from other ports. The temporarily installed port in [.filename]#/var/tmp/`make -V PORTNAME`# must be tested for proper operation to make sure there are no problems with paths. `PREFIX` must not be set explicitly in a port's [.filename]#Makefile#. Users installing the port may have set `PREFIX` to a custom location, and the port must respect that setting. Refer to programs and files from other ports with the variables mentioned above, not explicit pathnames. For instance, if the port requires a macro `PAGER` to have the full pathname of `less`, do not use a literal path of [.filename]#/usr/local/bin/less#. Instead, use `${LOCALBASE}`: [.programlisting] .... -DPAGER=\"${LOCALBASE}/bin/less\" .... The path with `LOCALBASE` is more likely to still work if the system administrator has moved the whole [.filename]#/usr/local# tree somewhere else. [TIP] ==== All these tests are done automatically when running `poudriere testport` or `poudriere bulk -t`. It is highly recommended that every ports contributor install and test their ports with it. See <> for more information. ==== [[testing-poudriere]] == poudriere For a ports contributor, poudriere is one of the most important and helpful testing and build tools. Its main features include: * Bulk building of the entire ports tree, specific subsets of the ports tree, or a single port including its dependencies * Automatic packaging of build results * Generation of build log files per port * Providing a signed man:pkg[8] repository * Testing of port builds before submitting a patch to the FreeBSD bug tracker or committing to the ports tree * Testing for successful ports builds using different options Because poudriere performs its building in a clean man:jail[8] environment and uses man:zfs[8] features, it has several advantages over traditional testing on the host system: * No pollution of the host environment: No leftover files, no accidental removals, no changes of existing configuration files. * Verify [.filename]#pkg-plist# for missing or superfluous entries * Ports committers sometimes ask for a poudriere log alongside a patch submission to assess whether the patch is ready for integration into the ports tree It is also quite straightforward to set up and use, has no dependencies, and will run on any supported FreeBSD release. This section shows how to install, configure, and run poudriere as part of the normal workflow of a ports contributor. The examples in this section show a default file layout, as standard in FreeBSD. Substitute any local changes accordingly. The ports tree, represented by `${PORTSDIR}`, is located in [.filename]#/usr/ports#. Both `${LOCALBASE}` and `${PREFIX}` are [.filename]#/usr/local# by default. [[testing-poudriere-installing]] === Installing poudriere poudriere is available in the ports tree in package:ports-mgmt/poudriere[]. It can be installed using man:pkg[8] or from ports: [source,shell] .... # pkg install poudriere .... or [source,shell] .... # make -C /usr/ports/ports-mgmt/poudriere install clean .... There is also a work-in-progress version of poudriere which will eventually become the next release. It is available in package:ports-mgmt/poudriere-devel[]. This development version is used for the official FreeBSD package builds, so it is well tested. It often has newer interesting features. A ports committer will want to use the development version because it is what is used in production, and has all the new features that will make sure everything is exactly right. A contributor will not necessarily need those as the most important fixes are backported to released version. The main reason for the use of the development version to build the official package is because it is faster, in a way that will shorten a full build from 18 hours to 17 hours when using a high end 32 CPU server with 128GB of RAM. Those optimizations will not matter a lot when building ports on a desktop machine. [[testing-poudriere-setup]] === Setting Up poudriere The port installs a default configuration file, [.filename]#/usr/local/etc/poudriere.conf#. Each parameter is documented in the configuration file and in man:poudriere[8]. Here is a minimal example config file: [.programlisting] .... ZPOOL=zroot BASEFS=/usr/local/poudriere DISTFILES_CACHE=/usr/ports/distfiles RESOLV_CONF=/etc/resolv.conf .... `ZPOOL`:: The name of the ZFS storage pool which poudriere shall use. Must be listed in the output of `zpool status`. `BASEFS`:: The root mount point for poudriere file systems. This entry will cause poudriere to mount `tank/poudriere` to `/poudriere`. `DISTFILES_CACHE`:: Defines where distfiles are stored. In this example, poudriere and the host share the distfiles storage directory. This avoids downloading tarballs which are already present on the system. Please create this directory if it does not already exist so that poudriere can find it. `RESOLV_CONF`:: Use the host [.filename]#/etc/resolv.conf# inside jails for DNS. This is needed so jails can resolve the URLs of distfiles when downloading. It is not needed when using a proxy. Refer to the default configuration file for proxy configuration. [[testing-poudriere-create-jails]] === Creating poudriere Jails Create the base jails which poudriere will use for building: [source,shell] .... # poudriere jail -c -j 131Ramd64 -v 13.1-RELEASE -a amd64 .... Fetch a `13.1-RELEASE` for `amd64` from the FTP server given by `FREEBSD_HOST` in [.filename]#poudriere.conf#, create the zfs file system `tank/poudriere/jails/131Ramd64`, mount it on [.filename]#/poudriere/jails/131Ramd64# and extract the `13.1-RELEASE` tarballs into this file system. [source,shell] .... # poudriere jail -c -j 12i386 -v stable/12 -a i386 -m git+https .... Create `tank/poudriere/jails/12i386`, mount it on [.filename]#/poudriere/jails/12i386#, then check out the tip of the Git branch of `FreeBSD-12-STABLE` from `GIT_HOST` in [.filename]#poudriere.conf# or the default `git.freebsd.org` into [.filename]#/poudriere/jails/12i386/usr/src#, then complete a `buildworld` and install it into [.filename]#/poudriere/jails/12i386#. [NOTE] ==== While it is possible to build a newer version of FreeBSD on an older version, most of the time it will not run. For example, if a `stable/13` jail is needed, the host will have to run `stable/13` too. Running `13.1-RELEASE` is not enough. ==== [NOTE] ==== To create a poudriere jail for `14.0-CURRENT`: [source,shell] .... # poudriere jail -c -j 14amd64 -v main -a amd64 -m git+https .... In order to run a `14.0-CURRENT` poudriere jail the host must be running `14.0-CURRENT`. In general, newer kernels can build and run older jails. For instance, a `14.0-CURRENT` kernel can build and run a `12.4-STABLE` if the `COMPAT_FREEBSD12` kernel option was compiled in (on by default in `14.0-CURRENT`[.filename]#GENERIC# kernel config). ==== A list of jails currently known to poudriere can be shown with `poudriere jail -l`: [source,shell] .... # poudriere jail -l JAILNAME VERSION ARCH METHOD 131Ramd64 13.1-RELEASE amd64 ftp 12i386 12.4-STABLE i386 git+https .... [[testing-poudriere-maintaining-jails]] === Keeping poudriere Jails Updated Managing updates is very straightforward. The command: [source,shell] .... # poudriere jail -u -j JAILNAME .... updates the specified jail to the latest version available. pass:[] For FreeBSD releases, update to the latest patchlevel with man:freebsd-update[8]. pass:[] For FreeBSD versions built from source, update to the latest git revision in the branch. [TIP] ==== For jails employing a `git+*` method, it is helpful to add `-J _NumberOfParallelBuildJobs_` to speed up the build by increasing the number of parallel compile jobs used. For example, if the building machine has 6 CPUs, use: [source,shell] .... # poudriere jail -u -J 6 -j JAILNAME .... ==== [[testing-poudriere-ports-tree]] === Setting Up Ports Trees for Use with poudriere There are multiple ways to use ports trees in poudriere. The most straightforward way is to have poudriere create a default ports tree for itself, using link:{handbook}mirrors/#git[Git]: [source,shell] .... # poudriere ports -c -m git+https -B main .... These commands create `tank/poudriere/ports/default`, mount it on [.filename]#/poudriere/ports/default#, and populate it using Git. Afterward it is included in the list of known ports trees: [source,shell] .... # poudriere ports -l PORTSTREE METHOD TIMESTAMP PATH default git+https 2020-07-20 04:23:56 /poudriere/ports/default .... [NOTE] ==== Note that the "default" ports tree is special. Each of the build commands explained later will implicitly use this ports tree unless specifically specified otherwise. To use another tree, add `-p _treename_` to the commands. ==== The best way to deal with local modifications for a ports contributor is to use link:{handbook}mirrors/#git[Git]. As with the creation of jails, it is possible to use a different method for creating the ports tree. To add an additional ports tree for testing local modifications and ports development, checking out the tree via git (as described above) is preferable. [[testing-poudriere-ports-tree-manual]] === Using Manually Managed Ports Trees with poudriere Depending on the workflow, it can be extremely helpful to use ports trees which are maintained manually. For instance, if there is a local copy of the ports tree in [.filename]#/work/ports#, point poudriere to the location: [source,shell] .... # poudriere ports -c -m null -M /work/ports -p development .... This will be listed in the table of known trees: [source,shell] .... # poudriere ports -l PORTSTREE METHOD TIMESTAMP PATH development null 2020-07-20 05:06:33 /work/ports .... [NOTE] ==== The dash or `null` in the `METHOD` column means that poudriere will not update or change this ports tree, ever. It is completely up to the user to maintain this tree, including all local modifications that may be used for testing new ports and submitting patches. ==== [[testing-poudriere-ports-tree-updating]] === Keeping poudriere Ports Trees Updated As straightforward as with jails described earlier: [source,shell] .... # poudriere ports -u -p PORTSTREE .... Will update the given _PORTSTREE_, one tree given by the output of `poudriere -l`, to the latest revision available on the official servers. [NOTE] ==== Ports trees without a method, see <>, cannot be updated like this and must be updated manually by the porter. ==== [[testing-poudriere-testing-ports]] === Testing Ports After jails and ports trees have been set up, the result of a contributor's modifications to the ports tree can be tested. For example, local modifications to the package:www/firefox[] port located in [.filename]#/work/ports/www/firefox# can be tested in the previously created 13.1-RELEASE jail: [source,shell] .... # poudriere testport -j 131Ramd64 -p development -o www/firefox .... This will build all dependencies of Firefox. If a dependency has been built previously and is still up-to-date, the pre-built package is installed. If a dependency has no up-to-date package, one will be built with default options in a jail. Then Firefox itself is built. The complete build of every port is logged to [.filename]#/poudriere/data/logs/bulk/131Ri386-development/build-time/logs#. The directory name `131Ri386-development` is derived from the arguments to `-j` and `-p`, respectively. For convenience, a symbolic link [.filename]#/poudriere/data/logs/bulk/131Ri386-development/latest# is also maintained. The link points to the latest _build-time_ directory. Also in this directory is an [.filename]#index.html# for observing the build process with a web browser. By default, poudriere cleans up the jails and leaves log files in the directories mentioned above. To ease investigation, jails can be kept running after the build by adding `-i` to `testport`: [source,shell] .... # poudriere testport -j 131Ramd64 -p development -i -o www/firefox .... After the build completes, and regardless of whether it was successful, a shell is provided within the jail. The shell is used to investigate further. poudriere can be told to leave the jail running after the build finishes with `-I`. poudriere will show the command to run when the jail is no longer needed. It is then possible to man:jexec[8] into it: [source,shell] .... # poudriere testport -j 131Ramd64 -p development -I -o www/firefox [...] ====>> Installing local Pkg repository to /usr/local/etc/pkg/repos ====>> Leaving jail 131Ramd64-development-n running, mounted at /poudriere/data/.m/131Ramd64-development/ref for interactive run testing ====>> To enter jail: jexec 131Ramd64-development-n env -i TERM=$TERM /usr/bin/login -fp root ====>> To stop jail: poudriere jail -k -j 131Ramd64 -p development # jexec 131Ramd64-development-n env -i TERM=$TERM /usr/bin/login -fp root # [do some stuff in the jail] # exit # poudriere jail -k -j 131Ramd64 -p development ====>> Umounting file systems .... An integral part of the FreeBSD ports build infrastructure is the ability to tweak ports to personal preferences with options. These can be tested with poudriere as well. Adding the `-c`: [source,shell] .... # poudriere testport -c -o www/firefox .... Presents the port configuration dialog before the port is built. The ports given after `-o` in the format `_category_/_portname_` will use the specified options, all dependencies will use the default options. Testing dependent ports with non-default options can be accomplished using sets, see <>. [TIP] ==== When testing ports where [.filename]#pkg-plist# is altered during build depending on the selected options, it is recommended to perform a test run with all options selected _and_ one with all options deselected. ==== [[testing-poudriere-sets]] === Using Sets For all actions involving builds, a so-called _set_ can be specified using `-z _setname_`. A set refers to a fully independent build. This allows, for instance, usage of `testport` with non-standard options for the dependent ports. To use sets, poudriere expects an existing directory structure similar to `PORT_DBDIR`, defaults to [.filename]#/var/db/ports# in its configuration directory. This directory is then man:nullfs[5]-mounted into the jails where the ports and their dependencies are built. Usually a suitable starting point can be obtained by recursively copying the existing `PORT_DBDIR` to [.filename]#/usr/local/etc/poudriere.d/jailname-portname-setname-options#. This is described in detail in man:poudriere[8]. For instance, testing package:www/firefox[] in a specific set named `devset`, add the `-z devset` parameter to the `testport` command: [source,shell] .... # poudriere testport -j 131Ramd64 -p development -z devset -o www/firefox .... This will look for the existence of these directories in this order: * [.filename]#/usr/local/etc/poudriere.d/131Ramd64-development-devset-options# * [.filename]#/usr/local/etc/poudriere.d/131Ramd64-devset-options# * [.filename]#/usr/local/etc/poudriere.d/131Ramd64-development-options# * [.filename]#/usr/local/etc/poudriere.d/devset-options# * [.filename]#/usr/local/etc/poudriere.d/development-options# * [.filename]#/usr/local/etc/poudriere.d/131Ramd64-options# * [.filename]#/usr/local/etc/poudriere.d/options# From this list, poudriere man:nullfs[5]-mounts the _first existing_ directory tree into the [.filename]#/var/db/ports# directory of the build jails. Hence, all custom options are used for all the ports during this run of `testport`. After the directory structure for a set is provided, the options for a particular port can be altered. For example: [source,shell] .... # poudriere options -c www/firefox -z devset .... The configuration dialog for package:www/firefox[] is shown, and options can be edited. The selected options are saved to the `devset` set. [NOTE] ==== poudriere is very flexible in the option configuration. poudriere can be set for particular jails, ports trees, and for multiple ports by one command. Refer to man:poudriere[8] for details. ==== [[testing-poudriere-make-conf]] === Providing a Custom [.filename]#make.conf# File Similar to using sets, poudriere will also use a custom [.filename]#make.conf# if it is provided. No special command line argument is necessary. Instead, poudriere looks for existing files matching a name scheme derived from the command line. For instance: [source,shell] .... # poudriere testport -j 131Ramd64 -p development -z devset -o www/firefox .... causes poudriere to check for the existence of these files in this order: * [.filename]#/usr/local/etc/poudriere.d/make.conf# * [.filename]#/usr/local/etc/poudriere.d/devset-make.conf# * [.filename]#/usr/local/etc/poudriere.d/development-make.conf# * [.filename]#/usr/local/etc/poudriere.d/131Ramd64-make.conf# * [.filename]#/usr/local/etc/poudriere.d/131Ramd64-development-make.conf# * [.filename]#/usr/local/etc/poudriere.d/131Ramd64-devset-make.conf# * [.filename]#/usr/local/etc/poudriere.d/131Ramd64-development-devset-make.conf# Unlike with sets, all of the found files will be appended, _in that order_, into one [.filename]#make.conf# inside the build jails. It is hence possible to have general make variables, intended to affect all builds in [.filename]#/usr/local/etc/poudriere.d/make.conf#. Special variables, intended to affect only certain jails or sets can be set in specialised [.filename]#make.conf# files, such as [.filename]#/usr/local/etc/poudriere.d/131Ramd64-development-devset-make.conf#. [[testing-poudriere-sets-perl]] .Using [.filename]#make.conf# to Change Default Perl [example] ==== To build a set with a non default Perl version, for example, `5.20`, using a set named `perl5-20`, create a [.filename]#perl5-20-make.conf# with this line: [.programlisting] .... DEFAULT_VERSIONS+= perl=5.20 .... [NOTE] **** Note the use of `+=` so that if the variable is already set in the default [.filename]#make.conf# its content will not be overwritten. **** ==== [[testing-poudriere-pruning-distfiles]] === Pruning no Longer Needed Distfiles poudriere comes with a built-in mechanism to remove outdated distfiles that are no longer used by any port of a given tree. The command [source,shell] .... # poudriere distclean -p portstree .... will scan the distfiles folder, `DISTFILES_CACHE` in [.filename]#poudriere.conf#, versus the ports tree given by the `-p _portstree_` argument and prompt for removal of those distfiles. To skip the prompt and remove all unused files unconditionally, the `-y` argument can be added: [source,shell] .... # poudriere distclean -p portstree -y .... + +[[testing-debugging-ports]] +== Debugging ports + +Sometimes things go wrong and the port fails at run time. +The framework provides some facilities to help in debugging ports. +These helpers are limited since the way of debugging a port heavily depends on +the technology used. +The following variables help with debugging ports: + +* `WITH_DEBUG`. If set, ports are built with debugging symbols. +* `WITH_DEBUG_PORTS`. Specifies a list of ports to be built with `WITH_DEBUG` set. +* `DEBUG_FLAGS`. Used to specify additional flags to `CFLAGS`. Defaults to `-g`. + +When `WITH_DEBUG` is set, either globally or for a list of ports, the resulting +binaries are not stripped. + +These variables can be specified in [.filename]#make.conf# or in the command +line: + +[source,shell] +.... +# cd category/port && make -DWITH_DEBUG DEBUG_FLAGSS="-g -O0" +.... + +[NOTE] +==== +If the port is built using package:ports-mgmt/poudriere[] the debugging +variables must be specified in poudriere's [.filename]#make.conf# and not in +[.filename]#/etc/make.conf#. +Refer to package:ports-mgmt/poudriere[] documentation for details. +==== + +Please refer to the debugging information in the +extref:{developers-handbook}[Developer's Handbook, debugging] for more details +about the debugging tools available.