Index: share/man/man7/tests.7 =================================================================== --- share/man/man7/tests.7 +++ share/man/man7/tests.7 @@ -26,118 +26,59 @@ .\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN .\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. .\" -.Dd April 13, 2014 +.Dd November 19, 2015 .Dt TESTS 7 .Os .Sh NAME .Nm tests .Nd introduction to the FreeBSD Test Suite .Sh DESCRIPTION -The -.Fx -Test Suite provides a collection of automated tests for two major purposes. -On one hand, the test suite aids -.Em developers -to detect bugs and regressions when they modify the source tree. On the other -hand, it allows -.Em end users -(and, in particular, system administrators) to verify that fresh installations -of the -.Fx -operating system behave correctly on their hardware platform and also to ensure -that the system does not suffer from regressions during regular operation and -maintenance. .Pp The .Fx -Test Suite can be found in the -.Pa /usr/tests -hierarchy. -.Pp -This manual page describes how to run the test suite and how to configure -some of its optional features. -.Ss Installing the test suite -The test suite is not yet installed by default as part of -.Fx , -but this is bound to change during the development of -.Fx 11.0 . -.Pp -If the -.Pa /usr/tests -directory is missing, then you will have to enable the build of the test -suite, rebuild your system and install the results. -You can do so by setting -.Sq WITH_TESTS=yes -in your -.Pa /etc/src.conf -file (see -.Xr src.conf 5 -for details) -and rebuilding the system as described in -.Xr build 7 . -.Ss When to run the tests? -Before diving into the details of how to run the test suite, here are some -scenarios in which you should run it: -.Bl -bullet -offset indent -.It -After a fresh installation of -.Fx -to ensure that the system works correctly on your hardware platform. -.It -After an upgrade of +Test Suite consists of many programs for testing the .Fx -to a different version to ensure that the new code works well on your -hardware platform and that the upgrade did not introduce regressions in your -configuration. -.It -After modifying the source tree to detect any new bugs and/or regressions. -.It -Periodically, maybe from a -.Xr cron 8 -job, to ensure that any changes to the system (such as the installation of -third-party packages or manual modifications to configuration files) do not -introduce unexpected failures. -.El -.Ss Running the tests -First, you will need to install the -.Sq devel/kyua -package from -.Xr ports 7 . -Then use the following command to run the whole test suite: +operating system. +These programs can be found under +.Pa /usr/tests. +.Sh RUNNING THE TESTS +The +.Xr kyua 1 +program must first be installed using +.Xr pkg 7 : +.Bd -literal -offset indent +$ pkg install kyua +.Ed +.Pp +To list all the tests under +.Pa /usr/tests : +.Bd -literal -offset indent +$ kyua list -k /usr/tests/Kyuafile +.Ed +.Pp +To run all the tests under +.Pa /usr/tests : .Bd -literal -offset indent $ kyua test -k /usr/tests/Kyuafile .Ed .Pp -The above will iterate through all test programs in -.Pa /usr/tests -recursively, execute them, store their results and debugging data in Kyua's -database (by default in -.Pa ~/.kyua/store.db ) , -and print a summary of the results. -This summary includes a brief count of all total tests run and how many of -them failed. -.Pp -It is possible to restrict which tests to run by providing their names in -the command line. -For example, this would execute the tests for the +To run tests only for .Xr cp 1 and -.Xr cut 1 -utilities: +.Xr cut 1 : .Bd -literal -offset indent $ kyua test -k /usr/tests/Kyuafile bin/cp usr.bin/cut .Ed -.Ss Obtaining reports of the tests execution -Additional information about the test results can be retrieved -by using Kyua's various reporting commands. -For example, the following would print a plain-text report of the executed -tests and show which ones failed: +.Sh REPORTING TEST RESULTS +After running the tests, a report can be generated +that shows the test output and lists the passed and failed tests. +.Pp +To generate a plain-text report: .Bd -literal -offset indent $ kyua report .Ed .Pp -This example would generate an HTML report ready to be published on a -web server: +To generate an HTML report: .Bd -literal -offset indent $ kyua report-html --output ~/public_html/tests .Ed @@ -145,7 +86,7 @@ For further details on the command-line interface of Kyua, please refer to its manual page .Xr kyua 1 . -.Ss Configuring the tests +.Sh CONFIGURING TESTS Some test cases in the .Fx Test Suite require manual configuration by the administrator before they can be @@ -164,25 +105,28 @@ .Bl -tag -width "allow_sysctl_side_effects" .It allow_devfs_side_effects If defined, enables tests that may destroy and recreate semipermanent device -nodes, like disk devices. Without this variable, tests may still create and -destroy devices nodes that are normally transient, like /dev/tap* and -/dev/pts*, as long as they clean them up afterwards. However, tests that +nodes, like disk devices. +Without this variable, tests may still create and +destroy device nodes that are normally transient, like /dev/tap* and +/dev/pts*, as long as they clean them up afterwards. +However, tests that require this variable have a relaxed cleanup requirement; they must recreate any devices that they destroyed, but not necessarily with the same devnames. .It allow_sysctl_side_effects Enables tests that change globally significant .Xr sysctl 8 -variables. The tests will undo any changes in their cleanup phases. +variables. +The tests will undo any changes in their cleanup phases. .It disks -Must be set to a space delimited list of disk device nodes. Tests that need -destructive access to disks must use these devices. Tests are not required to -preserve any data present on these disks. +Must be set to a space delimited list of disk device nodes. +Tests that need destructive access to disks must use these devices. +Tests are not required to preserve any data present on these disks. .It fibs -Must be set to a space delimited list of FIBs (routing tables). Tests that -need to modify a routing table may use any of these. Tests will cleanup any -new routes that they create. +Must be set to a space delimited list of FIBs (routing tables). +Tests that need to modify a routing table may use any of these. +Tests will cleanup any new routes that they create. .El -.Ss What to do if something fails? +.Sh REPORTING TEST FAILURES TO DEVELOPERS If there is .Em any failure during the execution of the test suite, please consider reporting it to the