Changeset View
Standalone View
share/man/man7/tests.7
Show All 20 Lines | |||||
.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL | .\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL | ||||
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE | .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE | ||||
.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS | .\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS | ||||
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER | .\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER | ||||
.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR | .\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR | ||||
.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN | .\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN | ||||
.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | .\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||||
.\" | .\" | ||||
.Dd April 13, 2014 | .Dd November 19, 2015 | ||||
.Dt TESTS 7 | .Dt TESTS 7 | ||||
.Os | .Os | ||||
.Sh NAME | .Sh NAME | ||||
.Nm tests | .Nm tests | ||||
.Nd introduction to the FreeBSD Test Suite | .Nd introduction to the FreeBSD Test Suite | ||||
.Sh DESCRIPTION | .Sh DESCRIPTION | ||||
.Pp | |||||
The | The | ||||
jmmv: Why is the explanation of the utility of the test suite gone? | |||||
.Fx | .Fx | ||||
Test Suite provides a collection of automated tests for two major purposes. | Test Suite consists of many programs for testing the | ||||
wblockUnsubmitted Not Done Inline Actionss/many/a collection of/ wblock: s/many/a collection of/ | |||||
On one hand, the test suite aids | |||||
.Em developers | |||||
to detect bugs and regressions when they modify the source tree. On the other | |||||
hand, it allows | |||||
.Em end users | |||||
(and, in particular, system administrators) to verify that fresh installations | |||||
of the | |||||
.Fx | .Fx | ||||
operating system behave correctly on their hardware platform and also to ensure | operating system. | ||||
that the system does not suffer from regressions during regular operation and | These programs can be found under | ||||
maintenance. | .Pa /usr/tests. | ||||
.Pp | .Sh RUNNING THE TESTS | ||||
The | The | ||||
.Fx | .Xr kyua 1 | ||||
Test Suite can be found in the | program must first be installed using | ||||
.Pa /usr/tests | .Xr pkg 7 : | ||||
bjkUnsubmitted Not Done Inline Actionsports are still a supported way of installing things. bjk: ports are still a supported way of installing things.
It's probably best to just way that kyua… | |||||
wblockUnsubmitted Not Done Inline ActionsWorth saying that "The devel/kyua port or package must be installed." though, so the reader knows where to find it. wblock: Worth saying that "The devel/kyua port or package must be installed." though, so the reader… | |||||
hierarchy. | .Bd -literal -offset indent | ||||
$ pkg install kyua | |||||
.Ed | |||||
.Pp | .Pp | ||||
This manual page describes how to run the test suite and how to configure | To list all the tests under | ||||
wblockUnsubmitted Not Done Inline Actions"in" rather than "under" makes more sense to me. wblock: "in" rather than "under" makes more sense to me. | |||||
some of its optional features. | .Pa /usr/tests : | ||||
.Ss Installing the test suite | .Bd -literal -offset indent | ||||
The test suite is not yet installed by default as part of | $ kyua list -k /usr/tests/Kyuafile | ||||
.Fx , | .Ed | ||||
but this is bound to change during the development of | |||||
.Fx 11.0 . | |||||
.Pp | .Pp | ||||
If the | To run all the tests under | ||||
wblockUnsubmitted Not Done Inline Actionss/under/in/ (as above) wblock: s/under/in/ (as above) | |||||
.Pa /usr/tests | .Pa /usr/tests : | ||||
directory is missing, then you will have to enable the build of the test | |||||
suite, rebuild your system and install the results. | |||||
You can do so by setting | |||||
.Sq WITH_TESTS=yes | |||||
jmmvUnsubmitted Not Done Inline ActionsWhy is the explanation of WITH_TESTS gone? jmmv: Why is the explanation of WITH_TESTS gone? | |||||
in your | |||||
.Pa /etc/src.conf | |||||
file (see | |||||
.Xr src.conf 5 | |||||
for details) | |||||
and rebuilding the system as described in | |||||
.Xr build 7 . | |||||
.Ss When to run the tests? | |||||
jmmvUnsubmitted Not Done Inline ActionsWhy is this whole section gone? jmmv: Why is this whole section gone? | |||||
Before diving into the details of how to run the test suite, here are some | |||||
scenarios in which you should run it: | |||||
.Bl -bullet -offset indent | |||||
.It | |||||
After a fresh installation of | |||||
.Fx | |||||
to ensure that the system works correctly on your hardware platform. | |||||
.It | |||||
After an upgrade of | |||||
.Fx | |||||
to a different version to ensure that the new code works well on your | |||||
hardware platform and that the upgrade did not introduce regressions in your | |||||
configuration. | |||||
.It | |||||
After modifying the source tree to detect any new bugs and/or regressions. | |||||
.It | |||||
Periodically, maybe from a | |||||
.Xr cron 8 | |||||
job, to ensure that any changes to the system (such as the installation of | |||||
third-party packages or manual modifications to configuration files) do not | |||||
introduce unexpected failures. | |||||
.El | |||||
.Ss Running the tests | |||||
First, you will need to install the | |||||
.Sq devel/kyua | |||||
package from | |||||
.Xr ports 7 . | |||||
Then use the following command to run the whole test suite: | |||||
.Bd -literal -offset indent | .Bd -literal -offset indent | ||||
$ kyua test -k /usr/tests/Kyuafile | $ kyua test -k /usr/tests/Kyuafile | ||||
.Ed | .Ed | ||||
.Pp | .Pp | ||||
The above will iterate through all test programs in | To run tests only for | ||||
.Pa /usr/tests | |||||
recursively, execute them, store their results and debugging data in Kyua's | |||||
database (by default in | |||||
.Pa ~/.kyua/store.db ) , | |||||
and print a summary of the results. | |||||
This summary includes a brief count of all total tests run and how many of | |||||
them failed. | |||||
.Pp | |||||
It is possible to restrict which tests to run by providing their names in | |||||
the command line. | |||||
For example, this would execute the tests for the | |||||
.Xr cp 1 | .Xr cp 1 | ||||
and | and | ||||
.Xr cut 1 | .Xr cut 1 : | ||||
utilities: | |||||
.Bd -literal -offset indent | .Bd -literal -offset indent | ||||
$ kyua test -k /usr/tests/Kyuafile bin/cp usr.bin/cut | $ kyua test -k /usr/tests/Kyuafile bin/cp usr.bin/cut | ||||
.Ed | .Ed | ||||
.Ss Obtaining reports of the tests execution | .Sh REPORTING TEST RESULTS | ||||
jmmvUnsubmitted Not Done Inline Actionsnit: The list of top-level sections that are allowed in a manpage is well-defined by mandoc. This is the reason why all these sections were actually sub-sections marked with .Ss, and I think this markup change should be reverted. jmmv: nit: The list of top-level sections that are allowed in a manpage is well-defined by mandoc. | |||||
Additional information about the test results can be retrieved | After running the tests, a report can be generated | ||||
by using Kyua's various reporting commands. | that shows the test output and lists the passed and failed tests. | ||||
For example, the following would print a plain-text report of the executed | .Pp | ||||
tests and show which ones failed: | To generate a plain-text report: | ||||
.Bd -literal -offset indent | .Bd -literal -offset indent | ||||
$ kyua report | $ kyua report | ||||
.Ed | .Ed | ||||
.Pp | .Pp | ||||
This example would generate an HTML report ready to be published on a | To generate an HTML report: | ||||
web server: | |||||
.Bd -literal -offset indent | .Bd -literal -offset indent | ||||
$ kyua report-html --output ~/public_html/tests | $ kyua report-html --output ~/public_html/tests | ||||
.Ed | .Ed | ||||
.Pp | .Pp | ||||
For further details on the command-line interface of Kyua, please refer | For further details on the command-line interface of Kyua, please refer | ||||
to its manual page | to its manual page | ||||
.Xr kyua 1 . | .Xr kyua 1 . | ||||
.Ss Configuring the tests | .Sh CONFIGURING TESTS | ||||
Some test cases in the | Some test cases in the | ||||
.Fx | .Fx | ||||
Test Suite require manual configuration by the administrator before they can be | Test Suite require manual configuration by the administrator before they can be | ||||
run. Unless certain properties are defined, the tests that require them will | run. Unless certain properties are defined, the tests that require them will | ||||
be skipped. | be skipped. | ||||
.Pp | .Pp | ||||
Test suites are configured by defining their configuration | Test suites are configured by defining their configuration | ||||
variables in | variables in | ||||
.Pa /usr/local/etc/kyua/kyua.conf . | .Pa /usr/local/etc/kyua/kyua.conf . | ||||
The format of this file is detailed in | The format of this file is detailed in | ||||
.Xr kyua.conf 5 . | .Xr kyua.conf 5 . | ||||
.Pp | .Pp | ||||
The following configuration variables are available in the | The following configuration variables are available in the | ||||
.Fx | .Fx | ||||
Test Suite: | Test Suite: | ||||
.Bl -tag -width "allow_sysctl_side_effects" | .Bl -tag -width "allow_sysctl_side_effects" | ||||
.It allow_devfs_side_effects | .It allow_devfs_side_effects | ||||
If defined, enables tests that may destroy and recreate semipermanent device | If defined, enables tests that may destroy and recreate semipermanent device | ||||
wblockUnsubmitted Not Done Inline ActionsNot yours, but... passive->active: and "could destroy" reads better than "may destroy". wblock: Not yours, but... passive->active:
s/enables/enable/
and "could destroy" reads better than… | |||||
nodes, like disk devices. Without this variable, tests may still create and | nodes, like disk devices. | ||||
destroy devices nodes that are normally transient, like /dev/tap* and | Without this variable, tests may still create and | ||||
wblockUnsubmitted Not Done Inline Actionss/may/might/ ("may" implies permission) wblock: s/may/might/
("may" implies permission) | |||||
/dev/pts*, as long as they clean them up afterwards. However, tests that | destroy device nodes that are normally transient, like /dev/tap* and | ||||
bjkUnsubmitted Not Done Inline ActionsI don't see a macro specific for device names, so please use .Pa for these two. bjk: I don't see a macro specific for device names, so please use .Pa for these two. | |||||
/dev/pts*, as long as they clean them up afterwards. | |||||
However, tests that | |||||
require this variable have a relaxed cleanup requirement; they must recreate | require this variable have a relaxed cleanup requirement; they must recreate | ||||
any devices that they destroyed, but not necessarily with the same devnames. | any devices that they destroyed, but not necessarily with the same devnames. | ||||
.It allow_sysctl_side_effects | .It allow_sysctl_side_effects | ||||
Enables tests that change globally significant | Enables tests that change globally significant | ||||
wblockUnsubmitted Not Done Inline Actionss/Enables/Enable/ (as above) wblock: s/Enables/Enable/ (as above) | |||||
.Xr sysctl 8 | .Xr sysctl 8 | ||||
variables. The tests will undo any changes in their cleanup phases. | variables. | ||||
The tests will undo any changes in their cleanup phases. | |||||
.It disks | .It disks | ||||
Must be set to a space delimited list of disk device nodes. Tests that need | Must be set to a space delimited list of disk device nodes. | ||||
wblockUnsubmitted Not Done Inline ActionsI would say "space-delimited". wblock: I would say "space-delimited". | |||||
destructive access to disks must use these devices. Tests are not required to | Tests that need destructive access to disks must use these devices. | ||||
preserve any data present on these disks. | Tests are not required to preserve any data present on these disks. | ||||
.It fibs | .It fibs | ||||
Must be set to a space delimited list of FIBs (routing tables). Tests that | Must be set to a space delimited list of FIBs (routing tables). | ||||
wblockUnsubmitted Not Done Inline Actions"space-delimited" wblock: "space-delimited" | |||||
need to modify a routing table may use any of these. Tests will cleanup any | Tests that need to modify a routing table may use any of these. | ||||
wblockUnsubmitted Not Done Inline ActionsDoes "may" here mean "are allowed to" or "might"? wblock: Does "may" here mean "are allowed to" or "might"? | |||||
new routes that they create. | Tests will cleanup any new routes that they create. | ||||
wblockUnsubmitted Not Done Inline Actionss/cleanup/clean up/ ("cleanup" is a thing, "clean up" is a verb) wblock: s/cleanup/clean up/
("cleanup" is a thing, "clean up" is a verb) | |||||
.El | .El | ||||
.Ss What to do if something fails? | .Sh REPORTING TEST FAILURES TO DEVELOPERS | ||||
If there is | If there is | ||||
.Em any failure | .Em any failure | ||||
during the execution of the test suite, please consider reporting it to the | during the execution of the test suite, please consider reporting it to the | ||||
.Fx | .Fx | ||||
developers so that the failure can be analyzed and fixed. | developers so that the failure can be analyzed and fixed. | ||||
To do so, either send a message to the appropriate mailing list or file a | To do so, either send a message to the appropriate mailing list or file a | ||||
problem report. | problem report. | ||||
For more details please refer to: | For more details please refer to: | ||||
▲ Show 20 Lines • Show All 41 Lines • Show Last 20 Lines |
Why is the explanation of the utility of the test suite gone?