This diff adds basic framework to write pure python atf-compatible tests.
Problem statement
OS exposes large hundreds of interfaces to interact with. Covering these interfaces with automated tests simplifies developer work and allows them to move fast.
Ability to write the tests cheaply is extremely important - sometimes just a tiny bit results to a written (or non-written) test.
Despite the fact that naive interface is C, writing tests in C may be a bit cumbersome. Setup/teardown wrappers and state introspection are a bit hard to implement in C. Shell support provide pretty good high-level interface, but is limited by the abilities of existing binaries - it is not possible to test specific system call invocation.
Python fills the gap between the two, providing great high-level interfaces and the ability to interact with C code. There are already some tests or test parts, that are written in python. However, without the native support of atf, test framework adopted by FreeBSD, each python test requires a shell wrapper, which is inconvenient.
This diff is a result of considering following intents:
- Adding tests should be easier (or at least the same level of complexity) as adding C or shell test
- Runner integration with shouldn't restrict framework abilities (think of test parametrisation, for example)
- Solution should keep atf/kyua interaction surface as tiny as possible to minimise the migration costs if other runner/test suite is considered
- Should allow to easily use/extend common FreeBSD-specific test helpers
This change implements basic atf support, along with some network-related helpers, allowing to easily build test cases.
Implementation
There are a number of implementation approaches that were considered. In order to run python test one needs to (1) pick a testing framework and (2) determine what is the interface to the current test runner, kyua. The former is important, as the framework is what developer actually interacts with. Having convenient functions and decorators simplifies writing tests. Rich introspection interfaces simplify troubleshooting. The latter is important as well - people and automation rely on kyua ability to store and show test stderr/stdout logs, along with the execution results.
Let's start with the framework first. Generally, python testing ecosystem is dominated by 3 popular testing frameworks - built-in unittest, pytest and nose2. The primary unitest benefit is that it exists out-of-the box in stock python. It has less features, leaving pytest and nose as the candidates. Pytest is _really_ popular - for example, its plugin list contains nearly 1k plugins. Its architecture is pretty flexible, allowing to implement the desired support as a module, leaving pytest as the framework of choice.
Current FreeBSD testing framework, kyua, is language-agnostic, which means it executes the tests as separate programs with pre-defined interface. Kyua engine support two such interfaces: TAP and its own, ATF, defined in atf-test-program(4) and atf-test-case(4). TAP is a well-know interface, supported by pytest. However, it interface is extremely simple - there are no test names, debug is limited to one line. It is not possible to pass any test requirements (require.user or timeout). Additionally, implementing "cleanup" logic would also be hard. Cumulatively, it lead to the evaluation and then selection of kyua proprietary protocol, ATF.
Glue all this together.
ATF three key things that are relevant to the design: (1) test listing is a separate test binary run with a special keyword. Kyua expects all test to be returned in a simple text format with key/value pairs. (2) kyua doesn't enforce any structure on stdout/stderr, it gather test result from the filename passed to the test binary. (3) Runner execute test cleanup procedure as a separate binary run, appending :cleanup postfix to the test name.
Implementation consists of the pytest plugin implementing ATF format and a simple C wrapper, which reorders the provided arguments from ATF format to the format understandable by pytest. Each test has this wrapper specified after the shebang. When kyua executes the test, wrapper calls pytest, which loads atf plugin, does the work and returns the result. Additionally, a separate "package", /usr/tests/atf_python was added to collect code that may be useful across different tests. It would be easier to illustrate with examples of listing/running the test to demonstrate all moving parts.
Listing the tests
kyua list -k /path/to/Kyuafile test_file.py calls /path/to/test_file.py -l.
First line of the file is filled by the code in atf_test.mk and looks like the following:
#!/usr/tests/atf_pytest_wrapper -S /usr/tests. Wrapper gets executed with '/usr/tests/atf_pytest_wrapper' '-S /usr/tests' '/path/to/test_file.py' '-l'
Wrapper adds -S /usr/tests to the PYTHONPATH so the tests can reference the aforementioned atf_python package. It then translates the arguments to pytest & plugin lingua:
pytest -p no:cacheprovider -s --atf --co /path/to/test_file.py. Pytest runs, looks into the /path and finds contest.py in /usr/tests, loading the plugin.
Then, pytest does the usual discovery (function names and classes, starting with test_ or Test prefixes).
For the listing, plugin simply suppresses default output (by removing "terminalreporter" plugin) and prints the list of plugins with their metadata. Metadata is gathered from the user-provided marks (@pytest.mark.require_user("root")).
Running the test
kyua debug -k /path/to/Kyuafile test_file.py test_test1 calls /path/to/test_file.py -r /tmp/status_file -s /test/path test_test1.
Wrapper gets executed with '/usr/tests/atf_pytest_wrapper' '-S /usr/tests' '/path/to/test_file.py' '-r' '/tmp/status_file' '-s' '/test/path' 'test_test1'.
Wrapper adds -S /usr/tests to the PYTHONPATH so the tests can reference the aforementioned atf_python package. It also constructs pytest nodeid from the test path and name so only this exact matching test is run.
The resulting rearrangement: pytest -p no:cacheprovider -s --atf --atf-source-dir /test/path --atf-file /tmp/status_file /path/to/test_file.py::test_test1.
Similar to the listing stage, pytest loads the plugin, performes the collection stage and then runs the test. ATF plugin attaches the report hook and translated pytest outcomes to the atf ones. At the terminal phase, plugin writes the test outcome to the supplied file and terminates.
Running cleanup
Cleanup is a biggest "impedance mismatch" between pytest and kyua. The former assumes python-only and really low-side-effect tests, thus promoting running the test cleanups in the same function as the test, after yield. It's also possible to write "teardown" method for the class if the test is class-based. The latter wants to execute cleanup via a separate binary call. This poses some complications with tests discovery.
For example, should the cleanups be seen as a separate tests? It turned out to be a bit problematic to generate the cleanup tests with the parametrised functions. Instead, current implementation patches runtest and setup/teardown hooks of the relevant test:
Cleanup part of kyua debug -k /path/to/Kyuafile test_file.py test_test1 calls /path/to/test_file.py -r /tmp/status_file -s /test/path test_test1:cleanup.
Wrapper notices ":cleanup" test postfix and constructs the following line:
pytest -p no:cacheprovider -s --atf --atf-source-dir /test/path --atf-cleanup --atf-file /tmp/status_file /path/to/test_file.py::test_test1. This is effectively the same as in the running the test, with the --atf-cleanup added. ATF plugin hooks at pytest_collection_modifyitems() and patches the tests so the run the cleanup procedure. The rest part is exactly the same.
What works and what doesn't
Works
- listing test, running tests, running cleanups
- Test result mapping (with the side remark that non-strict XFAIL maps to failed, as there is no such state in atf)
- Test metadata (implemented as marks, like @pytest.mark.require_user("root")) except X-‘NAME’ & require.diskspace
Doesn't
- Opaque metadata passing via X-Name properties. Require some fixtures to write
- -s srcdir parameter passed by the runner is ignored.
- No atf-c-api(3) or similar - relying on pytest framework & existing python libraries
- No support for atf_tc_<get|has>_config_var() & atf_tc_set_md_var(). Can be probably implemented with env variables & autoload fixtures
- Open questions
- Should the plugin exists as pytest-atf package? It'd love so, but I'm not sure on what's the best way to enforce usage of this plugin when installing tests & provide meaningful error when it doesn't exist.