Change Details

It introduces a new concept of an execution environment. By default this is current approach without changes, i.e. a test case is running within a so called host environment. A new execenv added to run a test case within a jail. A new metadata parameter called `execenv_jail` has been introduced to turn it on and configure a test case jail. The only default jail parameter provided by kyua is `children.max=16` -- not to bother about this obvious thing in every single test case. It can be overridden by a test case. The goal is to possibly improve the speed of the FreeBSD test suite by having options to run some tests in parallel without additional complexity on the test case side. For example, some network related tests already spawn jails to isolate many details but still have clashing jail naming, routing entries, etc. The example of how it could be used with atf-sh is ipfw divert tests, https://reviews.freebsd.org/D42314: Sequential run example: ``` /usr/tests/sys/netpfil/ipfw # time kyua test divert divert:in_div -> passed [14.652s] divert:in_div_in -> passed [6.867s] divert:in_div_in_fwd_out_div_out -> passed [9.283s] divert:out_div -> passed [14.998s] divert:out_div_out -> passed [6.741s] Results file id is usr_tests_sys_netpfil_ipfw.20231024-115507-667631 Results saved to /root/.kyua/store/results.usr_tests_sys_netpfil_ipfw.20231024-115507-667631.db 5/5 passed (0 failed) 68.66 real 0.20 user 1.41 sys ``` Versus parallel one: ``` root@taarch64:/usr/tests/sys/netpfil/ipfw # time kyua --variable parallelism=8 test divert divert:out_div_out -> passed [6.712s] divert:in_div_in -> passed [6.712s] divert:in_div_in_fwd_out_div_out -> passed [9.571s] divert:in_div -> passed [14.755s] divert:out_div -> passed [14.928s] Results file id is usr_tests_sys_netpfil_ipfw.20231024-115658-276215 Results saved to /root/.kyua/store/results.usr_tests_sys_netpfil_ipfw.20231024-115658-276215.db 5/5 passed (0 failed) 21.70 real 0.22 user 2.25 sys ``` All the related work like man updates or unit tests of the kyua itself are postponed for now, until this RFC comes to some agreement (or not). And from the process point of view the change should probably go through the vendor branch and/or official FreeBSD fork.

The goal is to possibly improve the speed of the FreeBSD test suite by having additional approach of test running in parallel without additional complexity on a test case side. For example, some network related tests already spawn jails to isolate many details but still have clashing jail naming, routing entries, etc. It introduces a new general concept of an execution environment. By default this is current approach without changes, i.e. a test case is running within a so called "host" environment. The second execution environment is added to run a test case within a temporary jail, it allows to isolate it more and run in parallel with other tests (of course, if conceptually it's possible for a specific test). The changes of Kyua/ATF interfaces, from bottom to top: 1 ATF based tests - The new `execenv` metadata property can be set to explicitly ask for an execution environment: `"host"` or `"jail"`. If it’s not defined, as all existing tests do, then it means `"host"`. - The new `execenv.jail` metadata property can be optionally defined to ask Kyua to use specific `jail(8)` parameters during creation of a temporary jail. An example is `"vnet allow.raw_sockets"`. 2 Kyuafile - The same new metadata properties can be defined on `Kyuafile` level: `execenv` and `execenv_jail`. - Note that historically ATF uses dotted style of metadata naming, while Kyua uses underscore style. Hence `execenv.jail` vs. `execenv_jail`. 3 kyua.conf, kyua CLI - The new `execenv` engine configuration variable can be set to a list of execution environments to run only tests designed for. Tests of not listed environments are skipped. - By default, this variable lists all execution environments supported by a Kyua binary, e.g. `execenv="host jail"`. - This variable can be changed via `kyua.conf` or via `kyua` CLI's `-v` parameter. For example, `kyua -v execenv=host test` will run only host-based tests and skip jail-based ones. - Current value of this variable can be examined with `kyua config`.

It introduces a new concept of an execution environment. By default this is current approach without changes, i.e. a test case is running within a so called host environmentThe goal is to possibly improve the speed of the FreeBSD test suite by having additional approach of test running in parallel without additional complexity on a test case side. A new execenv added to run a test case within a jail.For example, A new metadata parameter called `execenv_some network related tests already spawn jail` has been introduced to turn it on and configure a test case jail.s to isolate many details but still have clashing jail naming, The only default jail parameter provided by kyua is `children.max=16` -- not to bother about this obvious thing in every single test case.routing entries, It can be overridden by a test caseetc. The goal is to possibly improve the speed of the FreeBSD test suite by having options to run some tests in parallel without additional complexity on the test case sidIt introduces a new general concept of an execution environment. By default this is current approach without changes, i.e. For example,a test case is running within a so called "host" environment. some network related tests already spawn jails to isolate many details but still have clashing jail namingThe second execution environment is added to run a test case within a temporary jail, routing entriesit allows to isolate it more and run in parallel with other tests (of course, etcif conceptually it's possible for a specific test). The example of how it could be used with atf-sh is ipfw divert testschanges of Kyua/ATF interfaces, https://reviews.freebsd.org/D42314: Sequential run example: ``` /usr/tests/sys/netpfil/ipfw # time kyua test divert divert:in_div -> passed [14.652s] divert:in_div_in -> passed [6.867s] divert:in_div_in_fwd_out_div_out -> passed [9.283s] divert:out_div -> passed [14.998s] divert:out_div_out -> passed [6.741s]from bottom to top: Results file id is usr_tests_sys_netpfil_ipfw.20231024-115507-667631 Results saved to /root/.kyua/store/results.usr_tests_sys_netpfil_ipfw.20231024-115507-667631.db1 ATF based tests 5/5 passed (0 failed)- The new `execenv` metadata property can be set to explicitly ask for an execution environment: `"host"` or `"jail"`. If it’s not defined, as all existing tests do, then it means `"host"`. 68.66 real 0.20 user 1.41 sys ``` Versus parallel one: ``` root@taarch64:/usr/tests/sys/netpfil/ipfw # time kyua --variable parallelism=8 test divert divert:out_div_out -> passed [6.712s] divert:in_div_in -> passed [6.712s] divert:in_div_in_fwd_out_div_out -> passed [9.571s] divert:in_div -> passed [14.755s] divert:out_div -> passed [14.928s]- The new `execenv.jail` metadata property can be optionally defined to ask Kyua to use specific `jail(8)` parameters during creation of a temporary jail. An example is `"vnet allow.raw_sockets"`. Results file id is usr_tests_sys_netpfil_ipfw.20231024-115658-2762152 Kyuafile - The same new metadata properties can be defined on `Kyuafile` level: `execenv` and `execenv_jail`. Results saved to /root/.kyua/store/results.usr_tests_sys_netpfil_ipfw.20231024-115658-276215.db- Note that historically ATF uses dotted style of metadata naming, while Kyua uses underscore style. Hence `execenv.jail` vs. `execenv_jail`. 5/5 passed (0 failed)3 kyua.conf, kyua CLI - The new `execenv` engine configuration variable can be set to a list of execution environments to run only tests designed for. Tests of not listed environments are skipped. 21.70 real 0.22 user 2.25 sys- By default, this variable lists all execution environments supported by a Kyua binary, e.g. `execenv="host jail"`. ``` All the related work like man updates or unit tests of the - This variable can be changed via `kyua.conf` or via `kyua itself are postponed for now` CLI's `-v` parameter. For example, until this RFC comes to some agreement (or not). And from the process point of view the change should probably go through the vendor branch and/or official FreeBSD fork`kyua -v execenv=host test` will run only host-based tests and skip jail-based ones. - Current value of this variable can be examined with `kyua config`.