Change Details

This diff adds basic framework to write pure python atf-compatible tests. ## Problem statement OS exposes large hundreds of interfaces to interact with. Covering these interfaces with automated tests simplifies developer work and allows them to move fast. Ability to write the tests cheaply is extremely important - sometimes just a tiny bit results to a written (or non-written) test. Despite the fact that naive interface is C, writing tests in C may be a bit cumbersome. Setup/teardown wrappers and state introspection are a bit hard to implement in C. Shell support provide pretty good high-level interface, but is limited by the abilities of existing binaries - it is not possible to test specific system call invocation. Python fills the gap between the two, providing great high-level interfaces and the ability to interact with C code. There are already some tests or test parts, that are written in python. However, without the native support of atf, test framework adopted by FreeBSD, each python test requires a shell wrapper, which is inconvenient. This diff is a result of considering following intents: 1. Adding tests should be easier (or at least the same level of complexity) as adding C or shell test 1. Runner integration with shouldn't restrict framework abilities (think of test parametrisation, for example) 1. Solution should keep atf/kyua interaction surface as tiny as possible to minimise the migration costs if other runner/test suite is considered 1. Should allow to easily use/extend common FreeBSD-specific test helpers This change implements //basic// atf support, along with some network-related helpers, allowing to easily build test cases. ## Implementation There are a number of implementation approaches that were considered. In order to run python test one needs to (1) pick a testing framework and (2) determine what is the interface to the current test runner, `kyua`. The former is important, as the framework is what developer actually interacts with. Having convenient functions and decorators simplifies writing tests. Rich introspection interfaces simplify troubleshooting. The latter is important as well - people and automation rely on `kyua` ability to store and show test stderr/stdout logs, along with the execution results. Let's start with the framework first. Generally, python testing ecosystem is dominated by 3 popular testing frameworks - built-in unittest, pytest and nose2. The primary `unitest` benefit is that it exists out-of-the box in stock python. It has less features, leaving `pytest` and `nose` as the candidates. `Pytest` is _really_ popular - for example, its [plugin list](https://docs.pytest.org/en/7.0.x/reference/plugin_list.html) contains nearly 1k plugins. Its architecture is pretty flexible, allowing to implement the desired support as a module, leaving **pytest** as the **framework of choice**. Current FreeBSD testing framework, `kyua`, is language-agnostic, which means it executes the tests as separate programs with pre-defined interface. Kyua engine support two such interfaces: [TAP](https://testanything.org/tap-version-13-specification.html) and its own, ATF, defined in `atf-test-program(4)` and `atf-test-case(4)`. TAP is a well-know interface, supported by pytest. However, it interface is extremely simple - there are no test names, debug is limited to one line. It is not possible to pass any test requirements (`require.user` or `timeout`). Additionally, implementing "cleanup" logic would also be hard. Cumulatively, it lead to the evaluation and then selection of kyua proprietary protocol, **ATF**. Glue all this together. ATF three key things that are relevant to the design: (1) test listing is a separate test binary run with a special keyword. Kyua expects all test to be returned in a simple text format with key/value pairs. (2) kyua doesn't enforce any structure on stdout/stderr, it gather test result from the filename passed to the test binary. (3) Runner execute test `cleanup` procedure as a separate binary run, appending `:cleanup` postfix to the test name. Implementation consists of the pytest plugin implementing ATF format and a simple C wrapper, which reorders the provided arguments from ATF format to the format understandable by pytest. Each test has this wrapper specified after the shebang. When kyua executes the test, wrapper calls pytest, which loads atf plugin, does the work and returns the result. Additionally, a separate "package", `/usr/tests/atf_python` was added to collect code that may be useful across different tests. It would be easier to illustrate with examples of listing/running the test to demonstrate all moving parts. **Listing the tests** `kyua list -k /path/to/Kyuafile test_file.py` calls `/path/to/test_file.py -l`. First line of the file is filled by the code in `atf_test.mk` and looks like the following: `#!/usr/tests/atf_pytest_wrapper -S /usr/tests`. Wrapper gets executed with `'/usr/tests/atf_pytest_wrapper' '-S /usr/tests' '/path/to/test_file.py' '-l'` Wrapper adds `-S /usr/tests` to the `PYTHONPATH` so the tests can reference the aforementioned `atf_python` package. It then translates the arguments to pytest & plugin lingua: `pytest -p no:cacheprovider -s --atf --co /path/to/test_file.py`. Pytest runs, looks into the `/path` and finds `contest.py` in `/usr/tests`, loading the plugin. Then, pytest does the usual discovery (function names and classes, starting with `test_` or `Test` prefixes). For the listing, plugin simply suppresses default output (by removing "terminalreporter" plugin) and prints the list of plugins with their metadata. Metadata is gathered from the user-provided marks (`@pytest.mark.require_user("root")`). **Running the test** `kyua debug -k /path/to/Kyuafile test_file.py test_test1` calls `/path/to/test_file.py -r /tmp/status_file -s /test/path test_test1`. Wrapper gets executed with `'/usr/tests/atf_pytest_wrapper' '-S /usr/tests' '/path/to/test_file.py' '-r' '/tmp/status_file' '-s' '/test/path' 'test_test1'`. Wrapper adds `-S /usr/tests` to the `PYTHONPATH` so the tests can reference the aforementioned `atf_python` package. It also constructs pytest `nodeid` from the test path and name so only this exact matching test is run. The resulting rearrangement: `pytest -p no:cacheprovider -s --atf --atf-source-dir /test/path --atf-file /tmp/status_file /path/to/test_file.py::test_test1`. Similar to the listing stage, pytest loads the plugin, performes the collection stage and then runs the test. ATF plugin attaches the report hook and translated pytest outcomes to the atf ones. At the terminal phase, plugin writes the test outcome to the supplied file and terminates. **Running cleanup** Cleanup is a biggest "impedance mismatch" between pytest and kyua. The former assumes python-only and really low-side-effect tests, thus promoting running the test cleanups in the same function as the test, after `yield`. It's also possible to write "teardown" method for the class if the test is class-based. The latter wants to execute cleanup via a separate binary call. This poses some complications with tests discovery. For example, should the cleanups be seen as a separate tests? It turned out to be a bit problematic to generate the cleanup tests with the parametrised functions. Instead, current implementation patches `runtest` and `setup`/`teardown` hooks of the relevant test: Cleanup part of `kyua debug -k /path/to/Kyuafile test_file.py test_test1` calls `/path/to/test_file.py -r /tmp/status_file -s /test/path test_test1:cleanup`. Wrapper notices ":cleanup" test postfix and constructs the following line: `pytest -p no:cacheprovider -s --atf --atf-source-dir /test/path --atf-cleanup --atf-file /tmp/status_file /path/to/test_file.py::test_test1`. This is effectively the same as in the `running the test`, with the `--atf-cleanup` added. ATF plugin hooks at `pytest_collection_modifyitems()` and patches the tests so the run the cleanup procedure. The rest part is exactly the same. ### What works and what doesn't **Works** * listing test, running tests, running cleanups * Test result mapping (with the side remark that non-strict `XFAIL` maps to `failed`, as there is no such state in atf) * Test metadata (implemented as marks, like `@pytest.mark.require_user("root")`) except X-‘NAME’ & require.diskspace **Doesn't** * Opaque metadata passing via X-Name properties. Require some fixtures to write * `-s srcdir` parameter passed by the runner is ignored. * No `atf-c-api(3)` or similar - relying on pytest framework & existing python libraries * No support for `atf_tc_<get|has>_config_var()` & `atf_tc_set_md_var()`. Can be probably implemented with env variables & autoload fixtures ### Open questions * Should the plugin exists as `pytest-atf` package? It'd love so, but I'm not sure on what's the best way to enforce usage of this plugin when installing tests & provide meaningful error when it doesn't exist.

This diff adds basic framework to write pure python atf-compatible tests. ## Problem statement OS exposes large hundreds of interfaces to interact with. Covering these interfaces with automated tests simplifies developer work and allows them to move fast. Ability to write the tests cheaply is extremely important - sometimes just a tiny bit results to a written (or non-written) test. Despite the fact that naive interface is C, writing tests in C may be a bit cumbersome. Setup/teardown wrappers and state introspection are a bit hard to implement in C. Shell support provide pretty good high-level interface, but is limited by the abilities of existing binaries - it is not possible to test specific system call invocation. Python fills the gap between the two, providing great high-level interfaces and the ability to interact with C code. There are already some tests or test parts, that are written in python. However, without the native support of atf, test framework adopted by FreeBSD, each python test requires a shell wrapper, which is inconvenient. This diff is a result of considering following intents: 1. Adding tests should be easier (or at least the same level of complexity) as adding C or shell test 1. Runner integration with shouldn't restrict framework abilities (think of test parametrisation, for example) 1. Solution should keep atf/kyua interaction surface as tiny as possible to minimise the migration costs if other runner/test suite is considered 1. Should allow to easily use/extend common FreeBSD-specific test helpers This change implements //basic// atf support, along with some network-related helpers, allowing to easily build test cases. ## Implementation There are a number of implementation approaches that were considered. In order to run python test one needs to (1) pick a testing framework and (2) determine what is the interface to the current test runner, `kyua`. The former is important, as the framework is what developer actually interacts with. Having convenient functions and decorators simplifies writing tests. Rich introspection interfaces simplify troubleshooting. The latter is important as well - people and automation rely on `kyua` ability to store and show test stderr/stdout logs, along with the execution results. Let's start with the framework first. Generally, python testing ecosystem is dominated by 3 popular testing frameworks - built-in unittest, pytest and nose2. The primary `unitest` benefit is that it exists out-of-the box in stock python. It has less features, leaving `pytest` and `nose` as the candidates. `Pytest` is _really_ popular - for example, its [plugin list](https://docs.pytest.org/en/7.0.x/reference/plugin_list.html) contains nearly 1k plugins. Its architecture is pretty flexible, allowing to implement the desired support as a module, leaving **pytest** as the **framework of choice**. Current FreeBSD testing framework, `kyua`, is language-agnostic, which means it executes the tests as separate programs with pre-defined interface. Kyua engine support two such interfaces: [TAP](https://testanything.org/tap-version-13-specification.html) and its own, ATF, defined in `atf-test-program(4)` and `atf-test-case(4)`. TAP is a well-know interface, supported by pytest. However, it interface is extremely simple - there are no test names, debug is limited to one line. It is not possible to pass any test requirements (`require.user` or `timeout`). Additionally, implementing "cleanup" logic would also be hard. Cumulatively, it lead to the evaluation and then selection of kyua proprietary protocol, **ATF**. Glue all this together. ATF three key things that are relevant to the design: (1) test listing is a separate test binary run with a special keyword. Kyua expects all test to be returned in a simple text format with key/value pairs. (2) kyua doesn't enforce any structure on stdout/stderr, it gather test result from the filename passed to the test binary. (3) Runner execute test `cleanup` procedure as a separate binary run, appending `:cleanup` postfix to the test name. Implementation consists of the pytest plugin implementing ATF format and a simple C wrapper, which reorders the provided arguments from ATF format to the format understandable by pytest. Each test has this wrapper specified after the shebang. When kyua executes the test, wrapper calls pytest, which loads atf plugin, does the work and returns the result. Additionally, a separate "package", `/usr/tests/atf_python` was added to collect code that may be useful across different tests. It would be easier to illustrate with examples of listing/running the test to demonstrate all moving parts. **Listing the tests** `kyua list -k /path/to/Kyuafile test_file.py` calls `/path/to/test_file.py -l`. First line of the file is filled by the code in `atf_test.mk` and looks like the following: `#!/usr/tests/atf_pytest_wrapper -S /usr/tests`. Wrapper gets executed with `'/usr/tests/atf_pytest_wrapper' '-S /usr/tests' '/path/to/test_file.py' '-l'` Wrapper adds `-S /usr/tests` to the `PYTHONPATH` so the tests can reference the aforementioned `atf_python` package. It then translates the arguments to pytest & plugin lingua: `pytest -p no:cacheprovider -s --atf --co /path/to/test_file.py`. Pytest runs, looks into the `/path` and finds `contest.py` in `/usr/tests`, loading the plugin. Then, pytest does the usual discovery (function names and classes, starting with `test_` or `Test` prefixes). For the listing, plugin simply suppresses default output (by removing "terminalreporter" plugin) and prints the list of plugins with their metadata. Metadata is gathered from the user-provided marks (`@pytest.mark.require_user("root")`). **Running the test** `kyua debug -k /path/to/Kyuafile test_file.py test_test1` calls `/path/to/test_file.py -r /tmp/status_file -s /test/path test_test1`. Wrapper gets executed with `'/usr/tests/atf_pytest_wrapper' '-S /usr/tests' '/path/to/test_file.py' '-r' '/tmp/status_file' '-s' '/test/path' 'test_test1'`. Wrapper adds `-S /usr/tests` to the `PYTHONPATH` so the tests can reference the aforementioned `atf_python` package. It also constructs pytest `nodeid` from the test path and name so only this exact matching test is run. The resulting rearrangement: `pytest -p no:cacheprovider -s --atf --atf-source-dir /test/path --atf-file /tmp/status_file /path/to/test_file.py::test_test1`. Similar to the listing stage, pytest loads the plugin, performes the collection stage and then runs the test. ATF plugin attaches the report hook and translated pytest outcomes to the atf ones. At the terminal phase, plugin writes the test outcome to the supplied file and terminates. **Running cleanup** Cleanup is a biggest "impedance mismatch" between pytest and kyua. The former assumes python-only and really low-side-effect tests, thus promoting running the test cleanups in the same function as the test, after `yield`. It's also possible to write "teardown" method for the class if the test is class-based. The latter wants to execute cleanup via a separate binary call. This poses some complications with tests discovery. For example, should the cleanups be seen as a separate tests? It turned out to be a bit problematic to generate the cleanup tests with the parametrised functions. Instead, current implementation patches `runtest` and `setup`/`teardown` hooks of the relevant test: Cleanup part of `kyua debug -k /path/to/Kyuafile test_file.py test_test1` calls `/path/to/test_file.py -r /tmp/status_file -s /test/path test_test1:cleanup`. Wrapper notices ":cleanup" test postfix and constructs the following line: `pytest -p no:cacheprovider -s --atf --atf-source-dir /test/path --atf-cleanup --atf-file /tmp/status_file /path/to/test_file.py::test_test1`. This is effectively the same as in the `running the test`, with the `--atf-cleanup` added. ATF plugin hooks at `pytest_collection_modifyitems()` and patches the tests so the run the cleanup procedure. The rest part is exactly the same. ### What works and what doesn't **Works** * listing test, running tests, running cleanups * Test result mapping (with the side remark that non-strict `XFAIL` maps to `failed`, as there is no such state in atf) * Test metadata (implemented as marks, like `@pytest.mark.require_user("root")`) except X-‘NAME’ & require.diskspace **Doesn't** * Opaque metadata passing via X-Name properties. Require some fixtures to write * `-s srcdir` parameter passed by the runner is ignored. * No `atf-c-api(3)` or similar - relying on pytest framework & existing python libraries * No support for `atf_tc_<get|has>_config_var()` & `atf_tc_set_md_var()`. Can be probably implemented with env variables & autoload fixtures ### Open questions * Should the plugin exists as `pytest-atf` package? It'd love so, but I'm not sure on what's the best way to enforce usage of this plugin when installing tests & provide meaningful error when it doesn't exist.

This diff adds basic framework to write pure python atf-compatible tests. ## Problem statement OS exposes large hundreds of interfaces to interact with. Covering these interfaces with automated tests simplifies developer work and allows them to move fast. Ability to write the tests cheaply is extremely important - sometimes just a tiny bit results to a written (or non-written) test. Despite the fact that naive interface is C, writing tests in C may be a bit cumbersome. Setup/teardown wrappers and state introspection are a bit hard to implement in C. Shell support provide pretty good high-level interface, but is limited by the abilities of existing binaries - it is not possible to test specific system call invocation. Python fills the gap between the two, providing great high-level interfaces and the ability to interact with C code. There are already some tests or test parts, that are written in python. However, without the native support of atf, test framework adopted by FreeBSD, each python test requires a shell wrapper, which is inconvenient. This diff is a result of considering following intents: 1. Adding tests should be easier (or at least the same level of complexity) as adding C or shell test 1. Runner integration with shouldn't restrict framework abilities (think of test parametrisation, for example) 1. Solution should keep atf/kyua interaction surface as tiny as possible to minimise the migration costs if other runner/test suite is considered 1. Should allow to easily use/extend common FreeBSD-specific test helpers This change implements //basic// atf support, along with some network-related helpers, allowing to easily build test cases. ## Implementation There are a number of implementation approaches that were considered. In order to run python test one needs to (1) pick a testing framework and (2) determine what is the interface to the current test runner, `kyua`. The former is important, as the framework is what developer actually interacts with. Having convenient functions and decorators simplifies writing tests. Rich introspection interfaces simplify troubleshooting. The latter is important as well - people and automation rely on `kyua` ability to store and show test stderr/stdout logs, along with the execution results. Let's start with the framework first. Generally, python testing ecosystem is dominated by 3 popular testing frameworks - built-in unittest, pytest and nose2. The primary `unitest` benefit is that it exists out-of-the box in stock python. It has less features, leaving `pytest` and `nose` as the candidates. `Pytest` is _really_ popular - for example, its [plugin list](https://docs.pytest.org/en/7.0.x/reference/plugin_list.html) contains nearly 1k plugins. Its architecture is pretty flexible, allowing to implement the desired support as a module, leaving **pytest** as the **framework of choice**. Current FreeBSD testing framework, `kyua`, is language-agnostic, which means it executes the tests as separate programs with pre-defined interface. Kyua engine support two such interfaces: [TAP](https://testanything.org/tap-version-13-specification.html) and its own, ATF, defined in `atf-test-program(4)` and `atf-test-case(4)`. TAP is a well-know interface, supported by pytest. However, it interface is extremely simple - there are no test names, debug is limited to one line. It is not possible to pass any test requirements (`require.user` or `timeout`). Additionally, implementing "cleanup" logic would also be hard. Cumulatively, it lead to the evaluation and then selection of kyua proprietary protocol, **ATF**. Glue all this together. ATF three key things that are relevant to the design: (1) test listing is a separate test binary run with a special keyword. Kyua expects all test to be returned in a simple text format with key/value pairs. (2) kyua doesn't enforce any structure on stdout/stderr, it gather test result from the filename passed to the test binary. (3) Runner execute test `cleanup` procedure as a separate binary run, appending `:cleanup` postfix to the test name. Implementation consists of the pytest plugin implementing ATF format and a simple C wrapper, which reorders the provided arguments from ATF format to the format understandable by pytest. Each test has this wrapper specified after the shebang. When kyua executes the test, wrapper calls pytest, which loads atf plugin, does the work and returns the result. Additionally, a separate "package", `/usr/tests/atf_python` was added to collect code that may be useful across different tests. It would be easier to illustrate with examples of listing/running the test to demonstrate all moving parts. **Listing the tests** `kyua list -k /path/to/Kyuafile test_file.py` calls `/path/to/test_file.py -l`. First line of the file is filled by the code in `atf_test.mk` and looks like the following: `#!/usr/tests/atf_pytest_wrapper -S /usr/tests`. Wrapper gets executed with `'/usr/tests/atf_pytest_wrapper' '-S /usr/tests' '/path/to/test_file.py' '-l'` Wrapper adds `-S /usr/tests` to the `PYTHONPATH` so the tests can reference the aforementioned `atf_python` package. It then translates the arguments to pytest & plugin lingua: `pytest -p no:cacheprovider -s --atf --co /path/to/test_file.py`. Pytest runs, looks into the `/path` and finds `contest.py` in `/usr/tests`, loading the plugin. Then, pytest does the usual discovery (function names and classes, starting with `test_` or `Test` prefixes). For the listing, plugin simply suppresses default output (by removing "terminalreporter" plugin) and prints the list of plugins with their metadata. Metadata is gathered from the user-provided marks (`@pytest.mark.require_user("root")`). **Running the test** `kyua debug -k /path/to/Kyuafile test_file.py test_test1` calls `/path/to/test_file.py -r /tmp/status_file -s /test/path test_test1`. Wrapper gets executed with `'/usr/tests/atf_pytest_wrapper' '-S /usr/tests' '/path/to/test_file.py' '-r' '/tmp/status_file' '-s' '/test/path' 'test_test1'`. Wrapper adds `-S /usr/tests` to the `PYTHONPATH` so the tests can reference the aforementioned `atf_python` package. It also constructs pytest `nodeid` from the test path and name so only this exact matching test is run. The resulting rearrangement: `pytest -p no:cacheprovider -s --atf --atf-source-dir /test/path --atf-file /tmp/status_file /path/to/test_file.py::test_test1`. Similar to the listing stage, pytest loads the plugin, performes the collection stage and then runs the test. ATF plugin attaches the report hook and translated pytest outcomes to the atf ones. At the terminal phase, plugin writes the test outcome to the supplied file and terminates. **Running cleanup** Cleanup is a biggest "impedance mismatch" between pytest and kyua. The former assumes python-only and really low-side-effect tests, thus promoting running the test cleanups in the same function as the test, after `yield`. It's also possible to write "teardown" method for the class if the test is class-based. The latter wants to execute cleanup via a separate binary call. This poses some complications with tests discovery. For example, should the cleanups be seen as a separate tests? It turned out to be a bit problematic to generate the cleanup tests with the parametrised functions. Instead, current implementation patches `runtest` and `setup`/`teardown` hooks of the relevant test: Cleanup part of `kyua debug -k /path/to/Kyuafile test_file.py test_test1` calls `/path/to/test_file.py -r /tmp/status_file -s /test/path test_test1:cleanup`. Wrapper notices ":cleanup" test postfix and constructs the following line: `pytest -p no:cacheprovider -s --atf --atf-source-dir /test/path --atf-cleanup --atf-file /tmp/status_file /path/to/test_file.py::test_test1`. This is effectively the same as in the `running the test`, with the `--atf-cleanup` added. ATF plugin hooks at `pytest_collection_modifyitems()` and patches the tests so the run the cleanup procedure. The rest part is exactly the same. ### What works and what doesn't **Works** * listing test, running tests, running cleanups * Test result mapping (with the side remark that non-strict `XFAIL` maps to `failed`, as there is no such state in atf) * Test metadata (implemented as marks, like `@pytest.mark.require_user("root")`) except X-‘NAME’ & require.diskspace **Doesn't** * Opaque metadata passing via X-Name properties. Require some fixtures to write * `-s srcdir` parameter passed by the runner is ignored. * No `atf-c-api(3)` or similar - relying on pytest framework & existing python libraries * No support for `atf_tc_<get|has>_config_var()` & `atf_tc_set_md_var()`. Can be probably implemented with env variables & autoload fixtures ### Open questions * Should the plugin exists as `pytest-atf` package? It'd love so, but I'm not sure on what's the best way to enforce usage of this plugin when installing tests & provide meaningful error when it doesn't exist.